🚨 I NEED SOME ADVICE LADIES 🚨

Alright.. yes, it's about a guy. I know I know.. just bare with me. I was in a relationship with the same guy for 4 years. He was my first boyfriend, first kiss, first love, the guy I lost my virginity to, and my first (and second, third and fourth) heartbreak. So I'm not experienced with any other guys. But I decided to get a fresh start and move closer to my mother which is in a different state and it helped that I got away from my ex. And that's when I met HIM 😍

In October I started working at a restaurant that my mom also works at and she introduced me to some of her favorite people there, one of them being this guy that she's done nothing but say nice things about. He's cute, funny, sweet, and adores my mother which is so important to me. We talked at work, texted sometimes, and he even came over to hang out with me, my mom, and my moms boyfriend and had a few drinks. But he stopped working at the restaurant not too long ago so I never saw him after that. He randomly texted me on the 28th of December just to see what I was up to. I had some friends visiting so he invited us all over to his apartment. We had a great time. When we got home that night, he texted me and asked if I wanted to come back and stay the night with him. Of course I said no bc I had friends in town and didn't want to leave them bc that's rude lol he understood completely. On New Years Day we went back over to his house, but this time we all stayed over since we all decided to drink. We stayed up all night and he made my girls a bed in his brother's room (he shares the apartment with him and one of their friends) and he told me that I could sleep in his bed. So, of course, I'm thinking he's gonna come snuggle and try something. He didn't. I slept for an hour and a half before my friends came and woke me up to leave (I fought my sleep until 7:30 in the morning for this man 😭). When I wake up at home later that day I decide to kinda beat around the bush with him to see how he feels. Just little banter saying he should've come to lay with me and he says "yeahhh I just don't know about that yet" ??? So I play it off and say I'm totally cool with being friends and that I actually like having a friend here since my friends live 7 hours away. He then says "I really want to be friends with you and I want to have sex but I don't want to complicate things" (note: honesty appreciated). I told him I want the same thing, but whatever happens, happens. We're both grown but no matter what I really enjoy being his friend and I will be regardless. So he knows it's an open topic. I'm glad we're on the same page, but is it bad of me to still want for him to put his beautiful hands ALL OVER ME? I mean lol I hope we have sex and I know sex doesn't mean a relationship so I'm not worried about us having sex making things complicated. I'm more worried that me being around him and getting to know him will lead to me falling for him and that making things complicated. He already has a demeanor about him that I can see myself falling for. Whether or not we have sex, my only fear is to fall for him. But I don't want to stop anything from happening either. What do y'all think? Should I let it happen and see where things go or keep up my barrier to protect myself? HELP ME!!