Family member sick, SO still wants to visit

My SO's father has bronchitis and is pretty sick. His step sisters birthday party is tomorrow and it's at his fathers house. So, obviously his father and all the germs will be there. He JUST found out he has it tonight and has yet to start the antibiotics.. am I being a little nuts for not wanting to go to the party because I don't want to chance one of us getting sick/our 3 month old sick? I'm usually not a don't touch my kid I'm afraid of germs kind of person but he's sick enough were he was in the hospital all day today. I just don't want my 3 month old son to get sick. My SO seems to think it's fine and it's rude if we don't go to the party.