I feel like I'm going mad with worry .. herpes

My 10 week old son was seen by a doctor tonight who had a cold sore and did not wear gloves or wash his hands before examining him. He has been off his feeds and generally unhappy for 2 days. I asked the doctor about herpes virus and he said "nah it's nothing to worry about, I have a cold sore myself right now".... I feel sick with worry that my son is going to become ill now. The doctor said herpes in a baby is nothing to worry about, just viral and clears up in a few days.... I'm worried sick. Please tell me I'm crazy and nothing to worry about.