Accepting my infertility, guilt of not wanting to try anymore

While it's been a very bitter pill to swallow, in a few months time we'll have been TTC for four years. I cried a bit the other day, and a couple of weeks ago but my period started yesterday and I didn't cry at all. I think I am starting to accept my infertility and accept that I don't want to go to the doctors anymore and I don't want to try this anymore, etc.

But now that I am starting to accept it, I'm finding myself to have a lot of guilt. My husband knows I don't want to have investigative surgery, he knows I'm tired and disheartened. But I know I need to speak to him and see if he is happy to forget about this whole thing or if he still wants to keep trying.