21 week apt with doc.

Heather
This may be a bit of a rant but I'm confused n hurt! 
 I am 21 weeks tomorrow, the first 2 months I lost 10 lbs, and I am just now back up to by pre birth weight! I am not overly big, I believe I'm average! 170 lbs. at my apt today doc said he wants me to see a nutritionist because I gained 4 lbs this past month! 
What the heck, I am pregnant and not huge! No history of gestational diabetes in my prior pregnancies or family history! 
Wtf?! I feel like he was very rude about This and I'm kinda hurt! 
Anyone else deal with this?