Feeling selfish and confused

So, long story short, I've had a group of 4 girlfriends whom I was friends with in highschool and we all got pregnant at the same time 4 years ago. Fast forward to today, after petty drama and various other things, I decided to walk away from this group of women as it was becoming toxic to my life. Now, since one of the girls is dating my husbands brother, we still sometimes see this group of girls and their children. It's difficult because I see how much fun my daughter has with the other kiddos and remembers them but, I also want nothing to do with these women anymore.

Am I being a bad mother for not having my daughter around these other kids? I feel so guilty because she's so happy but, there's a lot of baggage with these women that I've chosen to leave. It's hard watching all of them play and knowing I'm the only one who doesn't see them anymore but, it was for the best.