To go to a party; to not go.... ?

Annelise • Mom of 5
I'm not sure what to do.... My son was invited to a birthday party for a friend of ours daughter. We were all planning on going, but I just found out that the girl who tried to convince my husband to cheat an leave me is going to be there. She's seriously nuts, he had to block her because she wouldn't get the hint. He met her a few times because she's friends with his cousins wife and ever since then she will message him things like "I'm so much better than your wife" and things about how he'd be happier with her. Just shit like that. Well now I don't wanna go to the party. I feel bad tho because this is my friends daughters 1st birthday. And ofcourse all my friends up here knew her way before they met me. So is it rude for me to take back our rsvp? Should I tell her why? This just sucks. This girl moved a few states away but ofcourse this is the weekend she decides to visit. :/