My baby won't drink his milk

My little guy is 4 months old. And used to drink 4-5oz of milk every 3 hours but recently he's only been taking 1-2 oz of milk every 3-4 hours... this is getting me increasingly anxious to the point i cry every time he refuses it...it gets me so frustrated with him I have to put him down and go cry in the bathroom... I really don't know what is wrong with him... He's been on the same formula since birth. And I'm really worried. Please help. GP just says he will drink when he's hungry but if that's the case he's never hungry unless it's 3am.