HELP!!! Losing everything to start a family?

Cutting a VERY long story short ish , I split with my husband nearly 20 months ago after he had an affair. We'd been together for 13 years since I was 15, married for 6 years, and had been trying to conceive for many years, suffering 2 miscarriages... We split when i found out he'd been having an affair with a mutual friend..

...I joined tinder almost immediately to boost my confidence, and within 3 weeks I met up with a guy who lives 2 hours drive away (basically just a hook up)... We ended up enjoying each others company and began seeing each other casually...which has developed and we have been in a fully fledged, official, in love relationship for over a year now, however without direction/plans...

...we have both known from the start that each of us ultimately wants a family and we now do want that with each other. I live this guy and I desperately want a family, as does he, so talks have turned to us moving in together where he lives (this would be his first home, as he currently lives with his mum-they run a farm together). Most of me wants to do this.. However...since my divorce I've focused on building myself and my independance: I own my own home (no problem I would rent this out I think), with a mortgage to pay and have a well paid (but stressful) job here, in a very small company in a position that was basically created for me and if I left they would have to replace me, there wouldn't be an option for me to return. There aren't many jobs where my guy lives, so it wouldn't be a case of pick and choose for position and salary... Equally, round here my options of meeting someone else and starting a family are slim(we both live in fairly rural areas).

Question is, should I give up everything I have built to start a life with this guy, or continue with my own life and hope that something else comes along one day, but it.might be too late (I'm 30). He's a tad older so I don't want to waste his time either, if this isn't happening we need to cut ties sooner rather than later to allow each other a chance to find this with someone else. Or is this our fairytale and we should put that above my career etc?

Note: he runs his own business that he wouldnt be able to move here... Life would be far simpler if he could!!!!

Feel like I'm having an early mid life crisis here... Help!!!!!!