Please don't bash! I just need to tell someone!

Okay, the story goes.... 
I'm married, 2 kids and all that jazz.
Been together going on 5 years and I love him, words can't describe how much. 
BUT! 
My first love, my first sexual partner got in contact. 
We broke up when I was 16, he cheated on me, ended up getting into a relationship with the girl he cheated on me with. She got pregnant. And I foolishly took him back when she was pregnant, then I got pregnant. It was a real messy situation, but the girl he cheated on me with told him a whole bunch of lies and he ended up not supporting me throughout my pregnancy, which I ended up miscarrying. 
Fast forward a bit, every relationship I've been in, I keep coming back to him. I will cheat on my partner because I know it's him I love. Now I have a husband and kids, i find myself on the same situation again. He's telling me he wants me back and an element of me wants to go there, hence why we've already slept together when we shouldn't have (should probably mention my husband isn't exactly all innocent himself) but I love my husband. But I also love my ex. I'm so confused on what to do. I will not leave my husband but I still want my ex.