Trying not to stress. Thought I was 6 weeks but only saw gestational sac

Caitlin • I`m a married 30 y/o high school digital integration specialist. My husband and I welcomed our rainbow baby into the world 1/2017 and are excited to prepare to expand our family in the next few months.
Because I have had 2 chemical pregnancies in the past, I an u/s today.  My doctor wanted to see me when I was about 6 weeks and based on due date calculators, we were gussing I am about 6 weeks and a few days.  However today, all that was seen was the gestational sac.  My doctor said that we could be only 4 or 5 weeks instead.  I got a positive hpt on 5/11 then 5/12 my HCG was 20 and went up to 128 on 5/16.  I go in for blood work on Thursday and Saturday to make sure my HCG is still increasing then I will have another ultrasound on Monday, 6/6 to see if there is anything new.  Should I worry yet?  This wasn't the news we were hoping for.  It isn't bad news, just not the little heartbeat we were hoping to see.