Anxiety about Rhogam shot

I have this unbelievably awful fear that them giving me this shot isn't in my best interest. I know I shouldn't listen to horror stories, but there have been women who listened to their doctors and ended up losing the baby "suddenly". I think this is more due to the fact that I have a panic disorder. I keep having nightmares that I'll get the shot and the next day my baby won't be moving and I lose her. I could never live with that. Is there any way to put my mind at ease? I especially don't trust my OB because he had my MIL taking hormones after a hysterectomy and told her to take them way longer than needed, and she ended up getting breast cancer from the hormone. I don't know what to do. I don't feel like I trust my OB. (I'm already switching doctors in that office by the way, but he's the one that said I needed the Rhogam shot without any explanation why) please help.