Fertility in men over 40

Meagan • TTC baby #2
I am a 30 yr old white female TTC #2, had my first at 25. My boyfriend is a 43 yr old black man whom has never had children. My daughters father is 10 yrs my senior too and it only took us two months trying to conceive. Is it cause I'm older now or could it be my boyfriends age? We've been trying three months now, hoping June is our month, I'm ovulating now best day is the third.... Really want to be pregnant and for him to have a child of his own..... Or is it too early to worry?