Sleep sex

Ok so has this ever happened to anyone? A lot of time by boyfriend fingers me in his sleep. I know he is asleep because I go hey what are you doing and he doesn't answer me. So I have to shake him awake and he says " whaaaaat?" & I go you're touching me! & he does nuh uh. The morning after it first happened I asked did you know you were doing this and he goes shut up not even. So he dead serious thought I was making it up. I have to admit now that he's aware he does this, sometimes I know he does on purpose. But other times I know he's asleep. Don't get me wrong I'm not complaining lol. But does any body else's SO do this?