Sleeping and he was touching me

So my boyfriend and I have been dating for two years now and he was over at my house after a wonderful movie date (where he was grabbing my butt and kissing me a ton ) and when he came over to my house we are watching another movie and he kept telling me to fall sleep because I was really tired and I had work the next day. So I fell sleep on his shoulder while he watched the movie and when I woke up it was around one in the morning. I woke up with my underwear kind of out of place and a wedgy. I didn't think anything of it until he told me what he said. And he said that he was fingering me while I was asleep. And for some reason I felt uncomfortable knowing that he was fingering me while I was asleep and unconscious. Honestly I wouldn't have thought it was a big deal if he was fingering me while waking me up like waking me up with some nice foreplay but it wasn't like that he wasn't trying to wake me up he just wanted to finger me even though I was asleep. Can I just please get some opinions on this   Is what he did wrong?