Cranky baby with me

My baby is 4 months old and I exclusively breastfeed him. I find that he gets crankier with me than he does when my husband is holding him. He will hand him to me and he will immediately start fussing...I try and feed him and he will eat for a minute or so then start cranking again. It's almost like he associates me witg eating and thinks he's hungry.....Anyone else with me.