Don't know what to do about riding?!?

I'm 6 weeks today, and can't decided if it's safe to keep riding my horses. It's not a new thing in my life, my body is use to it but it still makes me a little nervous because this is my first time being pregnant and I want to do what's best for my baby.... My doctor said its ok if I'm carful, but I'm just still nervous. Any other horse people out there?