Cold Sores and sex

Hey guys, So I've recently had my first cold sore ever and last night the guy I'm seeing and I were making out before he went down on me. So now I'm freaking out that I may have inadvertently transferred what's on my mouth through him to my vagina. Is that a thing?? If there was no contact from my the sore to any part of my body or his other than his lips, should I be worried? I'm super paranoid and that's making me stupid so any advice would be great!