Worried about toxoplasmosis please answer

Okay so my husband cleaned out the litter box last night and did that by flushing the cat poop in our toilet. He did that around 10 at night. Well I woke up around 3am and went and used the bathroom. I flushed the toilet before I peed because there was a little cat litter in the toilet. I didn’t wash my hands afterwards because I honestly didn’t think about it where I was half asleep. Anyways I sat down to use the bathroom and scratched the inside of my nose without thinking.

Now my paranoid brain is telling me that I could get toxoplasmosis from the toilet handle because I touched the inside of my nose. Do you think this is possible? Or do I have to have direct contact with cat poop?

Sorry if this sounds crazy. I’m half asleep still and worried. I go to the doctor on Thursday and I’m going to bring it up then too.