Just fussy?

So for about a week now my 3 week old goes through about 3 to 4 hours every day where he will cry if I try to put him down and sometimes he will cry even if i'm holding him or if he's in the middle of eating. Not sure if this is normal at this age or if he may be developing colic. Also he seems to calm down easier with his dad than with me, am I doing something wrong or is this just because he smells milk on me (we breastfeed) and not on his dad?