"Small" baby at 20 weeks means small baby at term?

My baby was measuring 5 ish days behind at our 20 week. Do you think this means he will be smaller when he is born? I want him to be helathy of course; but let's be honest, I will be pleased if he is on the smaller side...