Horseback riding while pregnant??

Hey horsey moms.. I have a wonderful Arab Mare and I've been riding my whole life, even went for a ride with a cast on my leg at one point, but now that I'm pregnant I feel very very hesitant to ride. I feel like my family is judging me for not exercising my horse enough but I feel like with so much at stake the risk of falling is just too high... 😕
My mare is 23 now and still in great shape, generally calm but loves to run and be silly. 
I'd like to hear any advice or experiences from other horse people out there... Am I being paranoid or is this really the best time to give my horse some time off?