Only gain one pound

So I am 21 weeks 3 days and I lost befor I gain but so far I only gain one pound and it's in the baby people look at me and ask me why ain't I big yet this is my 3nd baby but my doctor said he is not worry because I am a small person only 5"0 and I am 131lbs Should I be worry