Not NOT TTC with PCOS

I am 31 and recently diagnosed with PCOS. I have been with my boyfriend for 10+ years and we are very happy and comfortable with our life together. We decided several years ago that if we got pregnant it would be ok. While it was not something we were going to try for, it would be fine if it happened and we both would be happy. So for the last 4 years we have practiced the withdrawal method for bc. 
There have been enough "woops" incidents over the last few years that did not result in pregnancy to raise an eyebrow (I track my cycles). 
After my PCOS diagnosis, we are concerned but optimistic that it can happen eventually. Career wise I am not ready to actively pursue fertility treatments, but we have decided to stop making attempts to NOT get pregnant. 
Has anyone with PCOS just stopped all birth control and without activily TTC managed to get pregnant?
I should also mention that I am now on metformin for the PCOS. 
Thanks!!!