sex in our sleep?

I really hope I'm not the only one but  in the middle of the night when I'm in bed with my boyfriend I wake up and he's already fingering me. It's happened 3 times now and it's making me feel a little uncomfortable but I really don't want to hurt him, not that he enjoys pleasuring me while I'm asleep but he feels bad as well as its a surprise to him too. Like he'll just wake up and his hand is in my pants. Majority of the time o just go with it but the last time I was just too tired to keep going so I just nodded off and he kept going I don't even think he knew I was asleep. 
He means well and he loves me but it's just making me feel a little uncomfortable..
Has this happened to anyone else? Can anyone give us some advice on the situation?