Should I be concerned?

My lo is a small baby. At his 4 month visit he weighed 12 lbs and 11 ozs. He used to drink up to 29 oz of milk when he was about 3 months and his intake has slowly reduced to about 20 oz now. He refuses to drink unless he is too tired and sleepy. He is active, excited and playful all the time.He poops once a day and has about 5 wet diapers per day. I'm an concerned he is not drinking enough. My pediatrician says his weight gain is normal for a small baby. is anyone else's baby being fuzzy about drinking milk ?