Sleep sex

So, what I initially perceived as flattering for the first few years has now become annoying. My husband often starts to touch my breasts and vagina... gets all riled up and I used to always get aroused and end up sleeping with him and really enjoyed it. But now it’s annoying. I need my sleep because I work weird hours etc and need to have a strict routine for sleep and waking up to gym and run my other business so yeah. . . He doesn’t remember us having sex unless he cums and wakes up to help get us cleaned ☺️ otherwise he says he had a dream about having sex with me. But that also happens fairly often and he tells me about it and what not. But ja. What are your guys thoughts on this matter ?!