Is this a red flag ?

I went on a date last weekend it was the FIRST date I went on with the guy . We went out to eat then caught a movie because neither of us wanted to call it a night . He was scared to put his arm around me to I said "you can touch me I won't bite" We drank at the bar in the theater. I had a margarita and he had 2 shots neither of us were drunk . He drove me home it was a 45 min drive and it was almost 2 am I had fell asleep on the ride. When I woke up he was rubbing his hand between my thighs super close to my vagina under my skirt . I didnt say anything just moved his hand thanked him for the date he didn't try to come in or anything and I went inside .

I told him the next morning made me uncomfortable cuz he did that while I was sleep on our first date . We had only been texting a couple days and he tells me I said I told him he could touch me and thought I was flirting ....

I meant put his arm around me, or give me a hug NOT inappropriately .

Should I been more clear or was he just wanting to hear it that way ?

He wants to go on a date number 2 tomorrow I need help .