Cold Sores and Oral Sex

So my boyfriend periodically gets cold sores, and I don't. Since we're in a long distance relationship this isn't really an issue very often, but we'll be moving in together soon (!!!) and I realized I'm not exactly educated about the virus. Obviously refraining from contact when he has an outbreak, but is it possible to contract genital herpes from oral sex even when he doesn't have a cold sore? Sorry if that's a stupid question, I'm just trying to be as safe as possible.