Am i wrong?

ttc#2
OK I am a recovering alcoholic I've been sober for 5 months but my bf never drinks but wants to drink this weekend. I told him that I'm.not ready to be around alcohol and it would bother me. He got mad and started saying he feels trapped. I told him I would go to my sisters house while him and his friend drink and hangout. Am I wrong that I don't wanna be around alcohol? He tells me I'm not normal and to stop complaining.