Cats!

Need some advice from people with experience. I've always known pregnant women aren't supposed to change the litter box. My fiance has been doing it since we found out we are pregnant with our first. I went for my first appointment today and I am only about 5-6 weeks. I've had an indoor siamese for about 5 years. He follows me everywhere and even sleeps with me under the covers. So the nurse today kinda panicked when I told her about my cat. She said I need to stay away from him, vacuum the couches, sweep, vacuum floors, etc very often. Between 2 jobs and all this fatigue, I do not see myself doing that all the time! My cat scratches and meows all night if I don't let him in the bedroom. He is a huge cuddle bug and I cannot just kick him out. So my understanding of toxoplasmosis is that it is contracted through the stool. Do you think the nurse is correct with being so concerned with the fur? Idk if I should be more cautious or keep with my routine... anyone know more about this topic? Thanks and sorry for the long post!