Afraid for my relationship

So a little backstory. I am moving out of my apartment at the end of January (new company took over the properties and it's horrible now. My house and vehicle have been vandalized. They don't ever come out for repairs and the whole apartment complex has bug issues that they do nothing about) and me and my 2 Boys are moving in with my mother so I can save money and buy a house within the next 2 years but my issue is my mom lives an hour from my bfs house (I live 15 minutes from him and she's 45 minutes from my current apartment). I don't know how we're going to manage bc I already know we won't see each other often and I'm so worried it will affect our relationship. He has a 3 year old and I have a 6 & 2 year old so we only get to hang out 2-4 times a week if we're lucky and I already know once I move it will more then likely be far less then that and I'm so afraid our relationship will fall apart bc of it. His trailer is only a 2 bedroom so we can't move into together and he doesn't want to rent (his dad owns the property he's on) so we're going to wait and buy a house together but I really don't think we will last if we're not seeing much of each other. I told him this and he just kept saying we will make it work but I think once he's not getting the attention he's used to it will just start killing our relationship. I love him to death so this really bothers me. What can I do to stop with the negative thinking and what can I do to make sure our relationship is okay? Sorry for rambling. I feel stupid even coming on here complaining about this.