Sexsomnia

Victoria • 29 y/o. Diagnosed with PCOS 2014. Married 2012. Mortgage Underwriter.
This is a bit odd for me to admit but my husband has a slight case of sexsomnia. When I say "slight" I mean that he does have some memory of what occurred during the night but not how it happened. 
​ 
​It will be completely random. He'll change positions while sleeping (I know he's asleep bc of the snoring) and the next thing I know his hands are in my pants or he's tugging on them to get them off. I also know he's still asleep bc it's just...different. Like he's fumbling around or taking several breaks that last a little too long. Usually about halfway through the act he becomes lucid enough to keep going and finish. But each time he has no recollection of how it happened so I have to fill in the blanks.
​I can tell it kind of bothers him. It would upset me too if I were doing things in my sleep and have no memory of it. He's worried what would happen if I ever tell him no. He doesn't know what his mind will tell him to do if I don't let him have sex with me since that's what it wants him to do. I read that it's similar to sleepwalking and I've heard it's not good to wake someone in a harsh way. 
​The occurrences haven't been as frequent as they were when he was in his early 20's (he's 27) but it happened twice this week which is odd.
​Does anyone have any experience with this or know how I should handle it?