Cold Sores & Oral ?

So I get the occasional cold sore on my lip during the winter. I'm talking like, once every other year or so. I've not been sexually active in the past but now that I'm dating I wanted to try oral and did some research. I knew that it's very contagious and could become genital herpes via oral sex. So I obviously would steer clear if I had an active cold sore... But what if I don't have one? I've heard mixed advice from both doctors and websites. Like exactly 50/50. So now I'll ask you lovely people:

If I do NOT currently have a cold sore, is there still a risk of transferring the virus during oral?

The fact that I have doubt makes me want to be more safe than sorry but I also know cold sores are extremely common... And I don't typically hear horror stories about this scenario. Sooooo......? Share your thoughts