Fact or Fiction: Kitty Litter is Harmful to Unborn Baby

Hello! Curious to know if you think it's ok for a pregnant woman to change the kitty litter? I feel like I've read things online that say it's harmful, but then other sources say it's perfectly fine as long as you wash your hands afterwards. My husband isn't keen on doing it and I don't want to nag him about it every other day. Thoughts?