Advice please

I live with my parents and my grandmother lives next door with my uncle who is mentally handicapped. He has horrible hygiene issues and showers MAYBE once every few weeks (without soap) when my grandmother makes him and he’ll wear the same clothes for weeks on end. He never washes his hands or brushes his teeth at all ever. I just had a baby six weeks ago and he keeps getting his feelings hurt when I won’t let him touch or hold my baby because of his hygiene issues. My grandmother and my mom (his sister) have explained to him thoroughly why I won’t let him hold her and told him that if he would wash his hands or even use hand sanitizer that he could hold her but he refuses to do it. There’s no reasoning with him, he doesn’t understand that. I know the hygiene problem isn’t his fault and I feel bad for hurting his feelings but I just don’t feel comfortable letting him hold/touch my newborn, especially given the season. I don’t even want him to touch me and I have a great immune system, that’s how bad it is. What can I do? I hate hurting his feelings like this but there’s no way I could get him to clean up.