Refusing visitors

Maggie
My son is 3 weeks old today and people are constantly asking to come visit (mostly my parents, who live 2 hours away and have 5 other young children). It is flu season so any visitors make me feel uncomfortable, but for these past couple weeks we didn't have a choice but to let people visit since my fiancé's family flew in from Florida specifically to see him. He is extremely colicky all day every day, and I really don't like to have a bunch of visitors as I try and comfort him and help him calm down. So, I don't want visitors because of these reasons:  he is not even a month old, it is flu season and we live in Indiana where it is cold right now, and it is already difficult to go throughout my day on my maternity leave trying to handle his constant crying. Oh, and I'm getting the hang of breastfeeding and I would rather not have an audience. 
My family is making me feel bad for not letting them visit/not driving 2 hours to visit. Am I wrong for refusing to let them visit? They drove up the day he was born so they have met him.