Husband won't empty litter box even though I'm pregnant.

We have three indoor/outdoor cats.  He knows the chance of me getting toxoplasmosis from cleaning the litter box but he still won't do it for me.  He is gone at work 17 hours a day.  He leaves at 4:00am and doesn't get home until 9:00pm.  I stay home with our son so I'm home 24/7.  Is this something that would bother you or would you just let it go like I have?  I wear disposable rubber gloves every time and I always wash my hands when I'm done.