Feels like my Husband is avoiding TTC

A few months ago my husband and I had a conversation about TTC where we both agreed we were ready to try for one more baby. We were both serious about this, invested in opk and bbt testing, tracking my cycles, etc.

I was positive my husband genuinely was on board-- he has even been talking to our two other kids, asking how they feel about it and reminding them to be careful when horseplay/etc so if Mommy has a baby brother/sister we keep them safe. The problem is, our last two cycles it seems he has a reason to avoid sex on the most important days.

The week leading up to ovulation, he is all for it. We have lots of sex generally-- then the day before/day of/day after ovulation, he isnt interested. His body aches, or he is tired. This has happened two months in a row and while know we have both started working out in that same timeline (to be healthy for baby), it doesnt seem to bother him any other time of the month.

I dont want to be rude and ask if he has reservations and is avoiding sex on purpose, because I am well aware it could just be a coincidence. However, it really does disappoint me each time since we are getting older (37f/41m), and I have PCOS and he has ED (he takes meds for this) and diabetes so its already hard to enough to conceieve without missing the important dates.

Maybe I am putting too much pressure on us out of fear we wont be successful?