I don't like him

Soooo my boyfriend and I have our 4 month old together... and my bf has a twin brother that does heroin all the time.. I don't like him nor do I wanna be around him. On the 17th I have to go to a family gathering and he will be there... IM PISSED 
I don't wanna see him, talk to him or have him hold my son. What do I do!? I tell my bf how I feel and he tells me "my brothers never done anything to you" 
Fuck that .. he's a druggy and I DO NOT WANT THAT AROUND ME OR MY SON!! Help me !  
EDIT: I realize I don't have to be around him or speak to him but if my boyfriend has our kid and he just hands him to him, I can just see him doing that :( or anyone else in the family. Cause he already knows better not to ask me...