Sex and hsv. HELP!

I've recently been diagnosed with hsv 1 on my vagina (my bf ate me out when he had a cold sore, we thought it was a busted lip) I've been watching my OB as I should and all my blisters seemed to have disappeared. I could masturbate with two fingers just fine so I thought we would be able to have sex but it hurt really bad when we tried so we didn't. Could this be a blister on the inside I didn't feel? Help!