Im in Trouble..

Just got in trouble cause my friend got back with her ex. My SO doesn't like me around her ex. Her ex flirted with me(while me and my SO were broken up and him and her broke up we had sex as well.) I don't hang around either of them since then. My SO knows this. But yet he just yelled at me because they're back together... what the hell?