Baby not gaining weight

Okay so my LO was born at 40 weeks but his was only measuring 36 in my belly so technically a preemie, and he was born at 6 pounds 13 ounces but dropped down to 6 pounds 5 ounces over night and now is 5 weeks old and only 6 pounds 7 ounces, he had got jaundice really bad and dropped down to 5 pounds 5 ounces the first week and the doctor said he easnt worried that his weight would start to go up well he gained an ounce a day over 2 weeks and gained a pound back so he's now at 6 pounds 7 ounces and the doctor is now acting worried saying that depending on how much he weighs this week he might put him in the hospital but he has met all of his milestones and everything that I have read online has said that more than likely he is fine and personally I feel like he is completely fine and he's just a little baby since me and his father are both very small and had the same problem as children. but it's easier to lose weight than it is to gain as one doctor put it but now I'm worried about him being diagnosed with failure to thrive and Social Services trying to get involved and take him away I came across a horror story with someone in the exact same situation losing their son I personally think that he is fine because he's been getting his weight rapidly ever since his jaundice went away, but every baby is different he could just he a small baby like we both were, I was breastfeeding but I had to start supplementing to make sure he was getting enough so now i have him on a strict feedijg schedule of every 1.5 2 hours where I pump and then offer him that followed by a bottle of formula I have done EVERYTHING in my power to put weight on him and he's gaining just not quickly. And his family is gossiping that were not feeding him when that is NOT the case, I have done everyythinggg I can to continue to bring his weight up im constantly researching and have now found a specialist im going to call in the morning, it upsets me so badly becayse hes completely healthy aside from that, but people are judging me like I'm a horrible mother when I'm not. I was told I would never have children so I have been in aw ever since that first moment seeing him and have busted my ass to make sure he gains, So has anyone went threw thus with their preemie? Or at all? Please im losing my mind in worry!