Missed period, don't want to be let down.

When would you take a HPT? My husband and I are TTC #2, had sex every day last month, and I'm 3 days late. However, my husband and I both caught a cold (or something similar without nausea) so it's hard for me to differentiate between early pregnancy signs or just being sick. I had my IUD taken out 10/28 and we've been TTC since. I really don't want to be disappointed. When would you suggest I take a test?