Weight gain

Who knew the biggest struggle I would have during pregnancy was gaining weight. Had my appointment today at 20 weeks 3 days and I havent gained any weight yet, my midwife said I should have gained at least 5 pounds by now. Anyone else having difficulty gaining weight?