am I being too cautious?

I'm 13 weeks pregnant and my husband still thinks it's funny to startle me all the time. I absolutely hate it. we used to do it all the time before I got pregnant as a joke we would just out from a dark room when the other was leaving the bathroom and stuff like that. I would laugh it off but right now it just makes me uncomfortable. I'm worried it could be harming the baby when I get startled that way. I've tried telling this to him but he seems to always forget and he' says I'm just worried for nothing and it's all fun and games. am I being too cautious?? should I lighten up?