Baby's Percentile

My son's due date was Christmas day, but ended up being born 5 weeks early. He's on the small side for weight and height, but is on par with size if I were to go off his due date. Does this seem accurate? Should the doctor be looking at his size as if he were a 3 month old instead of a 4 month old? I keep forgetting to remind the doctor that he was born premature. He weights 10lbs 14oz, and is 23.5 in. He turned 4 months on March 24th.