Hands Off!

I'm not sure if there's a solution to this problem I'm experiencing, but any suggestions would be helpful.

My fiancé's quite the active sleeper. He sleeptalks regularly, tosses and turns often, and was known as an impressive sleepwalker as a kid in his family. Ever since we started living together several years ago, touching me in his sleep has been added to the above list of behaviors. I'd say at least once a week he grabs my butt, boobs, and fingers me... On really bad nights, he pulls out his dick and humps me, which ends up with a sleepy session of sex or me jerking him off🙄

Whenever this happens, I complain to him while he's awake (either after the session or the next morning) and he always says "sorryyy... but you're so hot/attractive/sexy/etc!" He does feel really bad and gross about it since he has no control over it, and it feels unfair that I'm the one losing sleep🥱 What can we do to prevent this from happening??