boyfriend wants to be closer to my family but

i dont want him too... my family is stupid and on drugs and so annoying... they used to abuse me when i was little so i dont want  him to be around them... I dont even get along with his family because they are all jealous of me.... !!  so it sucks for holidays i just want to be alone with him and spend it by ourselves but he always just wants to be at my familys house ... How can i tell him that not to be around my family anymore?😡im not a family person