Am I wrong for wanting to leave? Too much drama

My bf carries so much baby mama drama, this girl has came up with so much lies, just today my bf told me that she told his daughter not to be around me because I’m abusive towards children. I of course freaked out on him & told him I was done with this, she’s a “popular” girl & also has a big mouth so I know people are gonna start talking crap about me. Am I wrong for wanting to leave this relationship?