What point do you give up?

We have been ttc this month for 4 yrs. I haven’t been constant on tracking and all. But I’ve had all diagnostic testing and she put it down to ovulation and timing. I’m ovulating just not the same every month. During these 4 yrs I’ve never even had a false positive or anything. No implantation. We already have a 5 yr old. I had my first

IUI

this month. 6dpiui and I’m pretty sure it didn’t work. My husband is only allowing 2 more tries and says we should give up with treatment. This is month 4 of treatment. Do I just give up after this treatment or do I keep trying the next two. It just has me thinking If these past two didn’t work then maybe the next two won’t and what’s the point of wasting money. I’m not ready to give up, but at the same time I’m tired of disappointment. I’m so conflicted anymore.