On the verge of giving up

My LO is 3 weeks old and lately has been so fussy and niggly. He is constantly unhappy when with me - grunting and pushing to poop (not constipated - apparently this is NORMAL - who knew), and is so cranky when I hold him. The minute I hand him to someone else he is calm. Am starting to feel like this baby hates me and would honestly be better off without me. Can anyone relate?