New Due Date Worries..

I had my LMP the first week of December, I'm not sure of the exact date, & my periods come 3-7 days earlier each month. I found out I was pregnant the 1st week of January and my assumed due date was September 11th and that placed me at 8 weeks and 2 days today. I had my confirmation appointment today and per the ultrasound, the doctor placed me at only 5 weeks and 1 day. 3 weeks seems like such a big difference when it comes to the due date, so is this something I should be worried about regarding the baby's growth ? He says that everything looks good, but he went through everything so fast, & really didn't put my mind at ease, so I just don't know what to feel. Has this happened to anyone and the baby growth be okay ?