Midwives got it wrong...

Measuring my tummy since week 24 (I am now 30 + 6) - My midwives have lead me to believe that my baby is large just by measuring and feeling my baby boy. I was sent for a growth scan, which so happens to show the complete opposite. My baby is just about on the 10th centile, weighing at 3lb, so he is below the average and looking quite small for my gestational age. This is my second baby so they have taken absolutely everything into account - My height, weight and previous pregnancy etc. They have said it's obviously not a normal thing for me to be carrying so small and yet looking so damn big. My bump measures 35 weeks, yet my baby measures below. I now have to be monitored and have scans every two weeks to keep an eye on him. Anyone with this experience? Should I be as worried as I am? (It's currently 4am and I cannot sleep due to being restless!) Will they want him out early?