Partner seems he does not want anymore kids

Hi, I hope you are all well. 
I've been with my partner for 2.5years. 
He has two boys from his previous marriage. 
I'm 40 will be 41 in a couple of months time. He talked about having children 18months ago. We started to then try. 
Unfortunately we haven't been able to conceive. I've been to the doctors and have had all tests done and have all come back clear. Doctors  have said it's unexplained infertility and have suggested ivf. This is where my problem begins.....my partner doesn't want to try ivf. He says the live birth rates are not so great. We have had a lot of tests done privately as we don't qualify for the NHS service has he has kids already. 
On top of that since my fertile days last month he avoided dtd. We only managed to dtd twice during my fertile period. 
Today my fertile period has begun and he has not been near me at all. I feel upset and frustrated. I've talked to him and we are not getting anywhere. We are going away for a week on Thursday so hopefully we may get somewhere. 
His ex has been in touch in the last couple of weeks and has said she wants to move away with the kids in September. She wants to move 350 miles away from where we are. At the moment it's 45 miles away. 
Currently he sees his kids every Wednesday and every other weekend we have them at my place. He moved into my home as he gave the house to his ex for the kids. 
I know he is upset that he will not be able to see the kids weekly but it doesn't mean I should have to put my life on hold for him. 
I feel so hurt. 
Sorry for this being very long.