Kind of complicated?

I've been with my SO for 5yrs now, we have 2 kids together he is an awesome dad/partner in general awesome human been.

My so has been my first in everything even in kissing and, He's constantly telling me how much it means to him, and how much he loves me 4 that.

Okay here's the complicated part.

When we were boyfriend and girlfriend he broke up with me cuz he said I deserved better, I was mad cuz I didn't understand what he ment so trying to get back at him I had sex with some one else.

Months later we got back together, and told me he had sex with some girl while broken up so I also told him that I did to, but he didn't believe me he said I was just trying to hurt him for what he had done, I sworn to him and still didn't believe me..

When we got together I told him once more and he still didn't believe me, I haven't brought it up since then, and I feel very guilty that he still thinks he's been my only one, and I'm just scared that one day he finds out that what i told him is true, and leaves, and thinks I lied to him all this time we've been toghethr, even though I didn't he's the one that refused to believe.