Uterus measures small?

So I went to the doctor yesterday and I'm 24 weeks and when she measured my stomach she seemed worried because she said it was on the smaller side.. She said I need to come in again soon to get an ultrasound if it measures small again. I'm so worried.. What's going on? My health is in perfect condition and my baby's heart rate is fine. At my 20 week ultrasound he actually weighed over the average and was looking good. Please help I'm so paranoid