Extremely fussy baby

Anyone else have an extremely fussy baby? Not just during a “witching hour” literally all the time? My LO is going to be 3 weeks old and he is so fussy and I see other babies that can at least be content with just themselves for a little while but mine has to be in my arms or else he’s screaming. I want him to get use to being content in the swing or on the playmat. I have been cutting everything out of my diet thinking his stomach hurts, have given him formula and the formula seems to not hurt his stomach like my breastmilk does.