Am i a bad person?

So my bf Andy doesn't have trust for me.. I was at my ex's house (Landon) bc me and his sister are besties and I go there all the time, and my bf thought I was there cheating in him and wanting Landon back, and I wasn't there for Landon I was there for his sister! This is NOT the first time this has happened. So instead of dealing with Andy's bull crap any longer, I broke up with him! Is that too harsh to do if he doesn't trust me around my ex?