What am I doing wrong?

I have been with my husband for 4 years and never pressured the idea of having children. Until 8 months ago he told me he wanted to start trying to have a baby. So I agreed to try to have a baby because I too was ready just waiting for him to tell me when he was ready. I then started tracking my ovulation, taking my temperatures, lost 25 pounds, and going to the gym 8 hours a week plus doing the keto diet and taking prenatal vitamins every day. My husband and I only have sex 2 to 3 times a month if I'm lucky. He has such a low sex drive but wants a baby and when I tell him I got my period he gets sad as to why I'm not pregnant. I went to see my OBGYN who told me my progesterone level was low so she wanted to start me on a fertility medication called metformin to help me ovulate. When I told my husband what the doctor recommend he said to do whatever she said because we are ready. So my OBGYN prescribed me metformin for 4 months before doing major fertility meds like Clomid. When it comes time for us to have sex I don't tell him I'm ovulating because I don't want it to feel like I'm pressuring him or making it a chore. I just wear sexy lingerie and give him a b*** j** to seduce him but then he refuses sex because he's tired. At this point I'm so mentally and physically exhausted from even doing everything from losing weight, checking when I ovulate, checking my temperature every morning, and dealing with all the side effects from taking metformin plus I have to seduce him. I am doing all of the work to be rejected when it's the most important time in my cycle. I only get 4 months to try metformin and he just wasted a month by rejecting me. It makes me feel so sad knowing I'm doing all of this work for no gain plus I don't even get laid lol. It has me questioning myself as to why am I doing all of this if he is not truly gonna try and put forth any effort. I want to say I give up and let nature take its course but according to my OBGYN 40% of females have fertility issues. Which already put me in that category because I didn't ovulate with my regular 29 day cycle and it scares me to not do anything because the older we get the harder it is to conceive. Has anyone gone through this before because right now I feel like a failure and heartbroken.