Baby fever

So I'm feeling kinda disappointed. I had the IUD mirena removed almost 3 years ago. My ex and I tried for a baby for almost 6months with no avail. Now I'm in a New relationship and were not really preventing getting pregnant. We've been having unprotected sex for 2 straight months and he cums a lot inside me. But still no baby. I May also possibly have cystic fibroids and don't know if that would cause any problems. I have 3 previous kids and he has one. So I know we're both capable. I'm 32 and he's 36. Any advice.