Feeling discouraged

I had my IUD removed June 2019 and have been trying to get pregnant ever since using clomid and this month we switched to letrozole and af just arrived today 😔 I had an ectopic pregnancy in December that did set us back a bit but that is the closest we have gotten. 4 years ago it only took 2 rounds of clomid before I got pregnant with my now 3 yr old son. I just really want to give my son a sibling and have 1 more baby. But am I tempting fate? I’m starting to think this just isn’t meant to be and we should be happy with the great life we have now with our son. It is putting stress on my marriage and so disappointing every month. I can’t just “let it happen on its own” either as I do not ovulate without medication. I think I’ll give it a few more months but as we near the 1 year mark, I may have to give up...

Thanks for reading and letting me vent a bit