He makes me feel crazy!

So I'm almost 6 months pregnant, and my SO doesn't see an issue with going out one night every weekend to drink. He'll say he wants me to come along and gets mad when I don't want to. The only reason I don't want to is, he says it's only for a little bit but ends up being out way past midnight. So I'm too tired to be out that late and of course I get upset with him when he comes back that late because first he says he won't be late, and second because he's a parent too now. It's not fair that I have to sit at home and wait for him constantly, I feel like I'm his parent too sometimes. My question for you ladies out there is, am I stupid for being upset?