Am I wrong for this?

Am I wrong for not wanting to go to this dinner with my boyfriend bc his mom’s boyfriend will be there. He stares at my ass and is very creepy towards me and posts about the cameras and everything on Facebook of him always watching them in the house. No one in the family wants him there since he’s super aggressive and mean but the mom always invites him and I told my bf I’m not going if he’s there since the mom and bf have only been together 2 months me and my bf have been together 4 years and about to have a baby I feel like he shouldn’t even be invited he’s so mean to the family called the brother autistic to his face which I’m autistic and that was like a what the fuck.