Small baby?

OK mom's! Have any of you LOs always been smaller? Is it bad? Should I be worried? My son was born at 40 +3 but was too small for newborn he wore preemies for about two weeks then newborns til about 4 months. He's 10 months old now and wearing 6-9m but are still kinda long for him. He has gone from the 90th percentile to the 25th gradually since about 4 months. My husband think I'm over thinking but am I wrong for being worried. He also hits his milestones a bit late. Like right at the threshold were you should start worrieing about baby not doing something is when he starts

Age size.

NB. Pre

NB.-4m NB

4-6. 0-3

6-9. 3-6

10m. 6-9