First appointment showed baby 3 week younger then cycle predicted.

This is my second pregnancy. I lost my first last year. Yesterday I went in for my first appointment and ultrasound. According to my cycle I should have been 8weeks and 5 days. The ultrasound only showed 5 weeks. My dr is having me return in a week to do another scan. And was telling me that this happens a lot where the baby isn't as far along as predicted. After losing one this way I don't have much hope that this will turn out any different. Has anyone else had this happen and actually work out? Should I keep trying to have a baby after losing two?