Should I tell my husband I was raped?

Last night during sex my husband said the same phrase the rapist said when he raped me. I tried not to let it phase me, understanding my husband doesn't know about the rape. Afterwards I asked if he could please not say that phrase whenever we have sex. He was very nice and said of course. Then I started to tear up and he asked what was wrong. I wouldn't tell him and just said I was fine and I think he got irritated that I wouldn't tell him. 
The rape happened while he was in boot camp and I was in my first year of college. (A few years ago). We were dating then. Obviously I wasn't going to write him to tell him, and when I would see him that's not what I wanted to focus on. I just kept it buried inside. 
I'm conflicted... Should I tell him about it? I feel like he would get mad at me if he knew, or that he would blame me and want to leave. It makes me feel dirty and very ashamed, also because it happened when I went out with some girl friends to a club, so I guess I blame myself a little. I'm just so afraid of what he will say or think...but I hate keeping things from him. What should I do? Please help!