you do not know someone till you live with them

My boyfriend and his brother and I just got a house together. We have been good but except every time I talk about going to see my family he gets pissed off and tells me no. I do not like it one bit. Any suggestions?