How can I explain to my partner that when they are down it makes me feel down without it upsetting them?

My partner was having one of those days where he doesn’t know why he’s feeling the way he does so I immediately asked him if he was okay, he said he wasn’t and just said not to worry so I did but being me I could just feel the mood change and it was not helping the situation so I asked him if I could do anything to help and he just said no, I’m caring and can’t help but be concerned for him but it just upsets him to the point where he constantly says he shouldn’t of said anything because everything he opens up I just get mad or upset about how he’s feeling which isn’t true, I just get sad because I want to help but he just shuts down completely. I don’t know what to do honestly I feel like I’m at a loss😭 just to note* he said he didn’t know why he was feeling down but proceeded to tell me many reasons why he was feeling down… I’m only upset because I try to communicate with him how it makes me feel when he shuts me out when he feels this way but it just makes him upset