Baby worries


So I have been worried since leaving the doctor's office today. So I had an ultrasound done 3 weeks ago. The doctor told me that I was to early to see anything and was telling me I was only 5weeks and 4days... Went to the doctor's today and she told me that the baby was only measuring at 6 weeks and 3 days.. Should I be worried