Stuck in the middle

So i met my SO in the little town i grew up in. We have lived here for 3 1/2 years together. We just had a baby and all my family lives in this small town so it means a lot to me to be close to them. Well he grew up two hours away and all his family lives down there. We are stuck in the middle of chosing families pretty much. But we also want to move because his home town is a lot bigger which means better/more jobs for us. I really dont want to go because i don't want to leave my family. So we argue about it quite often. Im trying my hardest to be okay with going since he has spent a lot of time here with me and my home town, so i feel like i owe him this...