Kitty poo?

so me and my partner have started TTC, it's only been for a month so we're just starting out, however I've been told that cat poo can harm a fetus? How harmful is it? We have a little kitten that I'm very attached to. He follows me everywhere and never stops cuddling me. My partner reckons that it should all be okay once he only cleans the litter tray and I stay away from the cat? Or should I consider getting rid of my little kitty? Does anyone know?