Baby cries for daddy

My husband loves to hold his little baby boy now 4 weeks old every evening. But I feel terrible cuz our non fussy baby cries after just a little while being held by daddy. I try to tell him on how to hold him or what to do to see if he will stop. And he tries it. It will calm him for a minute or so and then he will fuss again. Last night my husband said that he just didn't like him and it made me feel so badly. Any ideas on how to get baby to bond better or what is the deal? He never really cries when I hold him.