Just stop trying?

I'm on the verge of stop trying for a baby. My husband and I have been trying for a baby for 3 years and nothing has happened. We have gone to the doctors and he has slightly low sperm count and I got diagnosed with PCOS about a year ago. Our doctor brought up IVF but we can't afford it. I'm 31. It's not like I'm getting any younger.