husband doesn't want sex so I flirt with other man

My husband and I have been married for 18 months together for 8years. We have been trying to conceive for over 2 years but the past year has been really difficult for me. I'm desperate for a baby but everytime I bring up having children and trying more to conceive he seems to not care. In the last 8 months we have slept together only once, I try to initiate sex and get the brush off I just feel so unattractive and unwanted. A couple of months ago I went on a work night out and a man who I have had a knack attraction for years have me some attention, he invited me to his hotel room which I laughed off at the time. Since then we have exchanged a few flirty texts and work calls between us have become more cheeky and flirty. I find him really attractive but know it's wrong and never dreamed of cheating before, at the minute tho I'm feeling really confused and secretly enjoy his attention. I know the right thing would be to stop completely and sort my problems with my husband but I'm scared if I ask him outright about the problems with our sex life hel say he doesn't want a family and our relationship won't survive????