Need serious advice

Tonight at dinner I was sitting with my boyfriend of three years high school best friends. Hubby was in the pool. We were discussing how we now have his daughter with a previous marriage full time. It got brought up that him and his ex wife used to cheat on each other like crazy. This was news to me as my boyfriend told me he’s only been with three other women and was faithful during their relationship. Basically they’re laughing saying who hasn’t he been with not knowing that I don’t know any of this. This honestly makes me feel different. I’m not sure what to think. I of course want to ask him, or honestly would love to sit with his Ex wife grab lunch and discuss . Because the story was completely different that I’ve heard. Recently there has been some behavior that has changed things for me a bit , but I just am in shock right now. Don’t know what to do