Horseback riding?

I just found out a week ago I'm pregnant, it's my first pregnancy by the way. I'm only 5weeks along. My husbands family is going horseback riding this weekend (which we usually do almost every weekend in the summer). We just go trail riding so I'm not jumping and bouncing around too much. And the horse I ride is very well tempered as she use to be brought to fairs and kids would ride her. So my question is, has anybody went trail riding this early? Should I just sit out? Could this hurt my growing baby? Cause me to miscarriage? (I mean it's always possible to get kicked or bucked off nothing is ever 100%) Just curious on your guys opinions, thanks!