Just want to vent about my nutty buddy

I've been occasionally having sex with a guy for about half a year now. It's not really a FWB thing, because we're not really friends. I caught feelings from the start (I'd liked him for a lot longer before we started having sex) but it's not going to be any more than this probably. We've never really talked about it because we both suck at talking about serious stuff, we just crack dumb jokes and insult each other when we're together.

I want to quit this because I've already let another guy pass (real relationship material). I just couldn't get him out of my head, being with him has become so comfortable. He was also my first so that probably makes it harder, even though I don't really care about that stuff consciously. My biggest problem is that I also have a high sex drive and it takes so much self restraint not to text him and go over to his place again 😬