Carefree sex

So I meet this guy a back in October and we both agreed to be "friends with benefits". From the beginning I knew what I was getting myself into. I wasn't expect in a relationship from and the same with him. We've had issues, he has come to me plenty of times with issues that he has had in regards to the custody problems with his kids. That was something that I wasn't ready to hear about at first. (The first mistake was not setting boundaries in what we decide to disclose to each other). So I let him vent caus e you never know what someone is really going through. I try my best not to disclose to much of my personal business with him. Last week on Valentine's Day I wasn't expecting nothing from him not even sex. He did text me to say he wanted to see me that night. Well I told him I was at dinner and left it at that. He didn't ask anything before and I told him that I would give him a ring when I got home that night. I did think he would care about what I was doing on V-day night. So when he came over he asked. I told him it was a dinner group thing and that's what it was and nothing else. I was little confused at why he wanted to know but kind of enjoyed that he was actually concerned. But we didn't "go there".
So you may wonder where is this going?!! Is the sex good? At first it wasn't nothing to brag about. Then it got better, then crazy to recently just flat out erotic! And I do enjoy it when it happens. My thing is I feel some type of way for letting myself really go there for a man that I could careless if we don't really end up with. I'm not pushing it and neither is he. I wish I could walk away from it all but the sex is what pulls me back into his trenches. We have argued and he even told me to my face that he is done with me. But a week later he comes back. We both whipped! I'm not sure what to do. And the funny thing I went a really long without sex and now I'm having it again I feel guilty/ but good about. Guilty cause the sex I'm having with him should be with the man I want to spend my life with. Good cause it just feels good. I don't know who has felt this way before?