Is my doctor lying?

I’m 24 weeks and only gained 11 pounds, I was at a good lower number on the BMI scale when I got pregnant, I have no health problems and feel great.. that being said my doctor today said because my first pregnancy was 8 lbs 12 oz, my next baby is going to be huge and I shouldn’t try to have a vaginal delivery because my pelvis might be too small. I feel very anxious and afraid now, it’s only one dr at the practice but what the hell!

Does this sound right? Why would my body make a baby too large to birth if I am not gaining or have blood sugar issues????