Feeling horrible! Hope I didn't hurt my baby

I'm a germaphobe and when I found out I was pregnant I did research on cleaning the cats litter box. I decided it would be safe for me to do it if I wore gloves/face mask and washed my hands afterwards. Well I've been grossed out by the face mask because the thought of the litter dust getting on the mask and than I touch the mask made me feel gross so I put the mask in the garbage. I started to change the litter box without a mask and just turned my head away when I dumped it but this time the wind blew the dust in my face!! I'm 24 weeks pregnant and I feel so horrible that I might of hurt my baby!! I called my doctor and waiting to hear back from her but how likely do you think it is I did any harm? Also my cat is an indoor cat only and only eats dry food. He was a stray cat as a kitten for maybe two months but we caught him, got his shots and everything and is healthy. We've had him for a year now and had him before I was pregnant.