Family

So I need your guys help. I love my family. They mean everything to me. I lived with my parents up until I was 19 and my current BF asked me to move into our own place. Now, because of him, I hardly ever see them. I haven't seen any of my family in over two weeks, despite living no more than 15 minutes away from them. When I expressed that I wanted to see my aunt for dinner tomorrow (she texted me and invited me over) he went off and said I needed to grow up and stop with "all the family shit" and that if "I wanted to see my family so bad, then to just go live with them again". I'm not sure what to do. He hates his whole family, so he never sees them unless he really wants to make an effort. I feel like because of this, he gets mad that I want to see my family. What do I do? I feel so hurt and upset. I miss my family. I want to see them, but I know it would just start a HUGE fight if I went over there