Separated parents who don't live near each other! I need advice!

I have a 3 year old with my highschool sweetheart. We haven't been together for a couple years now, and I have another baby due in June with someone else. I am currently living about 20 minutes away from my daughters dad and its worked out where he never has to drive up to my place, not that he would bother to anyway. I am always dropping her off because I work in his town. He has her every weekend friday until sunday morning, and a couple nights throughout the week for a few hours.

My current partner doesn't want to live anywhere near here- I would also love to move at least an hour away but I am anxious about separating my 3 year old from her dad.

My question is for parents who are in a similar situation- how do you do it? Do you have a week by week schedule? I know if I move it means one or both of us get less time with her. I don't want to spend a week or more without my baby :(

I also worry he would take me to court to gain custody back, right now I have 100% physical custody and we split legal. He has set visitation hours documented on paper from when she was 4 months old but we have since then developed our own schedule. He is ridiculously immature and I am so nervous about how he might react to me moving away within the next year or so..