Being sick is getting in the way of conceiving?

I've been TTC for a few months now, but every time that I try, I always end up sick near the time for AF. Whether it be the flu, viral infections, common cold, etc. I did just move half way across the country, and allergies are affecting me. But I feel like getting sick is getting in the way. Any opinions?