Help!!

So me and my bf have been dating for 4 years and he has only cane to my house about twice in those 4 years and has never meet my family like that or spent time wit them but he always wants me to be around him and his family..is he being unfair or am I stressing it..my family tells me I deserve better..should I stay or should I go!!!..and am moving soon and it’s about 40 min away from each other and he says it won’t work out if I move