Spilling the beans vent

I'm currently 5w3d. I don't see the doctor until next Tuesday and won't have an ultrasound until 3 weeks after that. My husband keeps telling people about the pregnancy. I'm okay with parents and siblings knowing, but he told 2 of my neighbors and random people at work. Whenever I tell him I wish he would stop, he says I'm fine and nothing bad's going to happen. That's easy for him to say.  It just makes me nervous.