Stress = Boyfriend doesn't want me

We have been together for almost 3 years. We have been stressed for almost 1 year now. Just financial stuff.. and whenever I go to hug him, he doesn't hug me back. He just stands there and continues talking. Or sometimes he tells me to "be serious for a minute". He never kisses me unless he is leaving for work. I always am the one to kiss his cheek or try to be close to him. I mean, he tells me he loves me often but I just don't feel it. Im sure things will get better when things change and start looking up. He is an affectionate man, just stressed right now, and irritated with me being so clingy I guess. But, again, if I tell him how I feel, he just gets defensive and lately in the mornings I have tried to avoid conversation because we have been having some intense arguments in the AM lately.