My chances at conceiving?

Im 27 years old, I was married for 9 1/2 years and in that marriage I conceived four beautiful babies. I've been divorced now for two years and I'm getting married again, my fiance and I have been trying for a baby for the last five months and nothing yet? I had a tumor when I was pregnant with my third baby and they had to remove that with my right ovary because test came back and I had cancer. But was able to get pregnant with my forth child a year and a half later. Wasn't trying for her but so blessed to have her lol. Could it be harder for me to conceive where it's from a different man? Or could it be because I'm a little older now and only have one ovary?