Having a hard time conceiving but I have kids?

Idk if anyone will respond, but is anyone else on here having a hard time conceiving with their partner but you both have previous children?? I feel like I'm losing my mind because we both have kids the youngest for ours are 2 and 3 (kids belong to different mother and mine have a different father) we having been trying steady for 6 months but and have been having unprotected sex for a year. Went to the ob today and my doctor didn't seem to give a crap and said I should be fine seeing as I had a child two years ago.. ugh if anyone has or is going through this feel free to voice your story! It's so frustrating and I feel like every month is just disappointing 😔 still hoping for the best though just don't know what the hold up is