I’ve been unknowingly spending the holidays with his exes family

My SO started bringing me around them last year. We all went to a winery & did tastings together. After that we’ve spent the 4th, Thanksgiving, and Christmas together. I’ve asked my SO when i first met them how he knew them and he said they’re all just “long time friends”. Today we were talking about our last time seeing them which was recent & I asked him again how he met the family & that’s when he told me they are the family of his ex. I was kind of in shock because I didn’t expect that. Like at all. He goes on to say his ex doesn’t really come around her family since moving to New York so I haven’t met her yet. But like wtf. Let me say, I do enjoy this family & they’ve all welcomed me with open arms but I just really wish I would’ve known the situation beforehand. I really want to know am I being upset for no reason or is this valid?? I kind of brushed it off when we talked about it, but obviously still bothered. I don’t want this to blow up into a whole argument, but I already have a history of acting insecure for nothing, so I need insight! I don’t think I’ll be attending anymore events or visiting anymore, but I’m sure my SO will. How do I handle this??