I need some advice

So I have known my husband for 3 1/2 years and we have been married for almost a year. About 6 months ago I moved back into my apartment with my kids. He is very controlling and we have bedroom issues. He can’t last longer than 3-5 minutes. It just really sucks. It hasn’t always been that way but it is now. When I moved out I met someone else and we ended up sleeping together. He found out and obviously throws it in my face all the time. I have def fallen out of love with him. But it’s to the point of he knows everywhere I go and is stalkerish. I really can’t see myself with him due to the fact of having slept with someone else and actually having an awesome sex life now. I just don’t see how any of this could even be repaired. Am I wrong for just wanting to give up and divorce him?