Is it weird to not want to have sex with my husband while he has a cold sore?

I freak out about the littlest things. I get cold sores too but I'm always so careful. I'm nervous that my husband will touch his mouth and then before sex touch me down there or himself and somehow transfer it so we get herpes or something! I'm in my fertile week so I really want to try for a baby. Am I freaking out over nothing? Should he not touch me down there at all? Obv washing his hands first if he did ofcourse but... Idk?!!!??