Don’t know what to do.

Hey guys, this topic always finds a way to cloud my mind. I had a boyfriend on and off for going on 3 years but throughout our relationship I’ll loose feelings and gain feelings for someone. When I’m dating other people it doesn’t happen. I don’t mean to hurt him or make him think he’s doing something because he doesn’t or hasn’t and I’ve tried to tell him that. I feel like our relationship struggle would end up being a lot of hard work for nothing. His parents don’t like me due to me being white and not of the Hispanic origin. And they don’t let me see him. I don’t want to cause more conflict in their family that I already have. I guess those factors help influence my feelings for him. Do you all think I’m in the wrong for leaving? Honest opinions and advice please.