should we quit trying?

So my husband and I have been together almost six hears and are ttc #1. I have been off BC for almost 3 years and on fertility meds for 2 months. We both have great paying jobs but seem to constantly be broke. I just got braces for my top teeth and found out I have to now have 2 teeth pulled and I had to get bottom braces which set us back a couple grand. I also just had Lasik 3 months ago which thank goodness I will have paid off on Tuesday. I have a super stressful job and so does my husband. He just found out that even with all the work he has put in during this past year he might not even get a bonus. He is really down and depressed. With us being broke, his job issues, my fertility issues, and everything seeming to pile up... should I back off from trying to conceive? I know he really wants a baby but I feel like now is not a good time but he wants to keep going. He said he could find another and better job by the time the baby comes and he wants me to quit my job. I don't know what to think.. I told him I wanted to wait until next year to try again but truth be told I don't want to stop trying. Am I being foolish and selfish? I know if we made some lifestyle changes our finances would be just fine... I'm just feeling lost... He doesn't want me working so much and his paycheck could sustain us. We both make more than enough to get by on our own. I just wish my heart and my mind could get on the same page. My mind is telling me to wait and my heart is scream to keep going... thoughts? please no negativity. I am already feeling bad.. I don't need to feel worse