TTC #4 without success (long)

Hi! I'm 34 (35 In March) and my husband is 37. We have been together for almost 18 years but married for 12. We have 3 daughters (16, 14, and 13). Yes i was very fertile! I have wanted a 4th since our youngest was 2! He would agree and then change his mind. This happened about 3 or 4 times. Each time my heart broke a little more as the girls got older. I had an IUD put in when my youngest was about 18 months. Fast forward to 6 years ago to when I found out I had a 12cm ovarian cyst that had shoved my uterus over to my left side. I had it removed 1/2012. Turned out it was an endometrioma. It ruptured as they tried to remove it and the ovary was too stretched out to safe. Bye bye right ovary! That's how I found out I had endometriosis. I had my IUD removed 2 weeks later. Fast forward another 4 years and they found another endometrioma on my left ovary. We decided to do surgery since I can grow them big and not have pain! I walked away with my ovary this time. I had not used any birth control for 4 years and did not get pregnant. My husband then decided in 3/2016 that we can TTC using any method except for IVF. We discovered I wasn't ovulating very well so we tried with Clomid for like 8 months with no baby. Then we tried Femara for 1.5 cycles. I stopped it as it gave me migraines. I was having problems with increased anxiety so I stopped all fertility meds. My doc said she thinks our only hope at this point is IVF which my husband will NOT agree to. So, I just continue to "try" naturally and hope and pray each month for a bfp. So far I just get my heart broken each month. Especially on the cycles that male me feel like I am pregnant. I work in an obgyn clinic so it gets hard seeing all the pregnant moms with their big round bellies! Anyways I just wanted to share my (long) story. Thanks for reading if you got this far!