Why sex has to be moral?

I’m really happy with my relationship but I really enjoy flirting or sex with other men. Why is this morally wrong when you know deep inside who you really love and they know it as well?