I feel like giving up

I’m trying to have a baby. But I don’t get to meet up with the man I’m trying to have a baby with as often as I hoped. We have no romantic interest in one another, we just want to have a child. We had discussed everything beforehand. I understand that he works two jobs but he originally made it sound like he would always have enough free time for us to try. We’ve tried once so far. I have tested since and I’m currently not pregnant. I’m just so frustrated that we haven’t tried more and I’m trying to decide if I should look for someone else. We also don’t live close to each other, which I think is part of the problem. I know I could try a sperm bank, but then I would have to go into my savings. And if I have to go to a sperm bank repeatedly, then it may possibly wipe out my savings. The point of my savings was to cover anything I might need in terms of baby furniture or other essentials. If going to a sperm bank wipes out my savings then I might as well forget it, since I’ll just be putting myself in way too much debt. I just feel like I’ve made a fool of myself. And I’ve been under so much stress lately that I know I’m going to have trouble conceiving. I did have a couple of people try to talk me out of this. One because he thought at this point in my life that it was too late to have a baby. And the other said I was being to naive about how I was choosing to do it (but in all honesty he only called me naive after I told him I wasn’t going to have sex with him). I don’t know if I have it in me to keep trying. A part of me wants to and the other part is telling me to accept life as it is and just give up. I feel so alone.