Starving son to death by breastfeeding?

I’d like to start off that I think my son is as healthy as he can be, but I keep being told differently. He was born 6lbs 8oz and 3 weeks early. He started out nursing every 3 hours or about 7-8 times a day. He’s now 5 months old and weighs 13lbs 2oz. However, he nurses every 4 hours and sleeps through the night which means he nurses 4 times a day. He also gets solids a couple times a day. Every time I go to his pediatrician he pushes formula saying my son doesn’t nurse enough and that with how little he nurses that I’m starving him. As a result his body is apparently eating at his muscles which will cause him to die or at least that’s what his doctor says. Yet his doctor can follow that statement by telling me he’s floored by how ahead my son is physically since he rolls over both ways, can sit up, & is trying to crawl. Plus I’m a little confused since he says he’s growing fine but he’s just not happy with the nursing frequency. Ugh! I feed my son when he wants to eat. That just happens to be about every 4 hours/4 times a day. He’ll let me know when he’s going through a growth spurt and up it by 1-3 more nursing sessions. I do follow his lead though so idk what I’m doing wrong or how to fix it because if I try to make him nurse more or even sooner he just turns his head away and smiles at me. I would really like to hear from anyone who had a little one that wasn’t a big nursed and was and is okay.