Newborn and Sick Visitor

I'm a FTM and I'm anxious about EVERYTHING! My aunt and uncle are in town and wanted to see my 3 day old, so I said yes because I only see them a few times a year. They show up and my uncle has a cold. Ugh. I made him stay away and he wasn't allowed to hold the baby.

Should I be worried? Next time, do I just kick the sick visitor out?? I tell everyone before they come over, please if you're sick, do not come over - clearly that didn't work this time.