need advice!!!

So I met a guy through my best friend this past May, at the time he was with his baby moms. We would talk and it was a very nice thing. Fast forward to October and they're not together anymore. I invited him to a Halloween get together and he came (he lives an hour away, so I thought he would've said no) he know I'm really into him and at one point we started talking and I told him I wanted a serious relationship. He was super honest with me and told me that at the moment he just wanted to chill, that he liked me and that I was an awesome person but he just wanted to take things slow since he had just broken up with baby moms.He asked me out a couple of weeks after that.  It's been about 2 months since then and ever since we had been talking non stop, we've gone out a couple of times (and I don't label them dates because we pay for our own things [idk but i felt that since he asked me out he should've payed the first time]) about 2 weeks ago the long phone calls started and I was excited cause I felt like we were making progress. Then one night we stayed up until 3 and well I was horny and initiated some phone sex. After that he's been a little bit distant. Like we'll still talk but not as much as before. I brought it up to him because he had gone a day without talking to me and it was driving me insane and I kept thinking about him just dropping me after he got what he wanted. He told me that, that wasn't it (apparently he was trying to get under my skin).  I really have no idea what to do, I have a gut feeling that if it does start going somewhere I might get hurt. I told him this and he said to follow my gut, that that didn't mean that there wouldn't still be a chance but to follow it. I really like him, I really don't know what to do.