Need help or advice.

So, long story short. My husband and I met in Northern Indiana. I'm from Michigan, hes from Indiana, and we moved to Tennessee where we got married and had a child. No family down here. We have been here 4 years and are expecting our 2nd child. We both love it here and the opportunities here for us are endless. His family is narcissistic and they want him to move home and now he says he misses them and wants to go back home. Not only will they very negatively influence him but theres not many jobs for us there and we have bought a house here already. My family has said depending on what we do they would either follow us or visit quite often. Part of me feels guilty but yet since day 1 of my first child life, everything has been a fight with his family so I feel like living closer will be way more stressful for me. They smoke heavily and wont wash their hands or change their clothes. They feed him whatever they want even after I say no. They come and stay for weeks at a time without invitation and I constantly have to wipe pee of the toilet seat or clean up their trash and such, so you can see why I wouldnt want them too close to where they just show up without warning.

Anywho, they make him feel guilty about living so far away, about 7 hours of drive time, and it's starting to sink in.

What is your advice? Do you think I'm wrong for not wanting to go back? What to do?