Partner in love with someone else?!?!

My partner & I have been together for quite a few years now. I have some family friends I have known for years, one is transgender and her partner is a gorgeous young women. So they wanted to have a baby and we suggested my partner as a sperm donor. First go it worked, she fell pregnant and now has a one and a half year old gorgeous boy whom I have a lot to do with as they were close family friends. Anyway, I started having suspicions my partner formed feelings for the baby mama and I didn’t really say or do anything as I wanted too worried. He then told me he “wishes he could have slept with her” and I asked him about it this morning before I dropped him at work and all he did was RUB MY BACK! RUB MY BACK?!?! Didn’t say anything!!!! What do I do, how am I suppose to feel? Help!