jumps me when I'm sleeping

So me and my partner are fast asleep and all of a sudden he's hands are everywhere usually I tell him st stop and he just turns around and he never remembers the nxt day ..... And if I'm in the mood I let it carry on and he  sort of wonders how this started.... Does this happen to anyone else or