Exes with Benefits?

I recently got into a sexual relationship with an ex. We dated for a year and a half but things didn't end well. We broke up a year ago and it seems that many of the issues we had have been resolved (my clinginess, his drug abuse, distance, etc.) However, neither of us want a real relationship but we can't seem to stay away from each other. So now we have random hook ups. We are friends as well. But every time we hang out it always ends with something physical. Which is fine, because I enjoy it and I know he does too. As far as I can tell we like things this way but the last couple of times we got together, he has been extremely tender and sweet. He has accidentally said "I love you" several times but I just brush it off. What is going on? How do I handle this? How can I figure out how he feels when I don't even know how I feel?