SO hates my family

My baby's father hates my family and I don't know if I should be with him? He literally cannot stand my mother and it makes me feel like I'm always trapped in the middle. Hes constantly saying how much he doesnt like how my family acts and how we have no common sense and trys to make himself seem better than my family. Sometimes I even feel like he belittles me to make himself feel better.... I don't know what to do its a very complicated situation for me... Advice please!!