Should I just forget about it

I have been TTC for 9 years now. The only thing I haven't done is ivf. Doctors keep telling me to loose weight so I can have more tests done and so I can have ivf or just the plain fact that loosing weight might help. My bmi is 35 btw. Iam not massive and already lost 4 stone. I'm getting really fed up. I can't loose anymore weight I've tried. Is it my fault I can't get pregnant cause I'm not trying hard enough? Do I just do nothing and if it happens it was meant to be or do I do all I can to make it happen?