Relationship Falling Apart

My baby is 9 weeks old. He's the absolute love of my life. I've never been happier. Other than when his Dad is around. We had a great relationship up until we came home from the hospital. Now I can't seem to do or say anything right. Just now our son was crying when he was holding him and he says it's my fault that he doesn't like him. That I don't let him spend enough time with him. Couldn't possibly be that he was overly tired. Or your own fault that you spend the majority of your time outside smoking instead of with your family. Doesn't seem to matter what I do. It's wrong. Just feeling a little bad. Still loving my little guy more than anything.