Ugh lacking the “D”

My husband is 37(38 in May) and for the last 3 years he rarely wants to have sex. I thought maybe we got married and started to try for a baby last year he would maybe be into it more but I’m lucky if I get it once every two weeks. He was upset with me because sex was only on my terms( well I’m ovulating so I’m sorry but yes this is kinda based on my body). And then other times I when I’m not ovulating I still have to beg. Today was a low point. I’m in my window, I didn’t tell him because maybe it’s performance anxiety, and it’s been over a week and he isn’t in to it. Saids it’s not his fault but won’t(too lazy)make a doctors appointment to see what’s wrong. It’s been a year of trying and turns out I might have PCOS so if I am ovulating it it’s kinda time to try. I’m sitting here getting upset because I’m tired of the it’s not my fault. Blaming everything but himself and his body. I couldn’t even enjoy the little bit of sex we were able to have back it was too much with me moving against him. I can’t help but to think it’s me. I’ve always been a bigger girl. And I’ve lost almost 20 pounds and I’m planning on losing 10 more for my first mini goal of 30 and aim for 50 by June. I don’t really know what to think anymore. He saids it is me but it’s not his fault. I guess I just want to vent.