Having an affair while on IFV meds.

I met some one a few months ago and connected with him in many levels that I cannot explain. My husband of 6 years is wonderful and amazing but after 6 IUIs and 2 failed IVFs, I am unable to continue my physical relationship between us. Currently, I am on <a href="https://glowing.com/glow-fertility-program">IVF</a> #3. This man that I met has been in my life for about 4 months and I've been physically and mentally attracted to him. I still love my husband and don't intend to leave him. I need to know what to do and how to fix my relationship.
All advices welcome. 
Thanks