35 years & TTC

In July of 2017 me & my boyfriend decided that we was gonna try for a baby (I had 3 he had none). So I got off my birth control nexplanon and we started trying. We got engaged that month and got married in August of 2017. In September 2017 I was told that I had endometriosis and fibroids on my cervix. After 4 months of trying I found out I was pregnant with baby #4 in December 2017. Babygirl was born in August 2018. We have been trying for baby #5 since December 2018. I have an appt on the 15th of this month to find out what's going on. I guess my question is should I be more worried about me being 35 or the endometriosis? I have never had a problem with getting pregnant before and I'm just really upset that it's not happened yet. I really don't want to 40 and having another baby. Anybody got an advice