Weight

I'm 24 weeks pregnant 4 days. Every time I go to the doctor I either lose weight or remain the same since before I got pregnant, my doctor seems sorta worried but will visit again next week. Have checked my weight at home and has went down again. At my last visit(2 weeks ago), my baby almost weighed a pound. Should I be worried of anything or any complications!? Freaking out and don't know if I should be!