Fussy baby during mid day

Hi guys.  I'm a first time mom, my baby is 1week and 2 days young now. During morning and night time we have no problem waking him up and feeding him every 2 hours and getting him back to sleep, but for some reason around midday when he needs to be woken up to be fed he won't go back to sleep and keeps crying when you you try to put him down when you think he fell asleep, that could go on for about 4 hours he would be fussy, sleepy, crying and just wants to be carried it seems, but if he doesn't nap then he goes ballistic. What could it be? Any advice would help.