Never been into casual relationships

Ive never been interested in having a casual relationship. I was dead set on being in a relationship in my 20's, i waited to lose my virginity at 26 and by my 30's regretted not having more fun. Im now in my mid 30's and I'm friends with a guy i dated about 7 years ago. He wanted more intimacy back when we dated,but i was very shy. He ended up getting married,having a kid and now he's divorced.I've been single for 3 years,i have kids and pretty much vowed to concentrate on my kids until they're older. Me and this guy have become quite close over the last 2 months or so,we talk everyday and then it turned into phone sex. I had no intention of meeting up with him, but one day i invited him over. I expected to be very shy and didn't anticipate anything sexual happening due to past dating experiences. I felt really comfortable with him,we flirted, talked alot and out of nowhere we did everything but have sex.I'm still processing it all because I've never had a casual thing before.he's still talking to me as usual,but Not as much. I know he has his son this week,so I've kept my distance for the time being. I'm worried i may start having feelings for him and then getting hurt.I know its too late now 😅 this is more of a venting post to be honest.I'm trying to play it cool but at the same time I'm wanting to talk to him alot.