Weight gain/ Loss

I gained 4 lbs a couple weeks ago and when I went back I lost weight again I'm 24 weeks and yet they aren't saying much my doctor seemed a little concerned. But since high school my weight always fluctuated like this. Anyone else just can't gain weight yet my baby boy is healthy and growing.