Consent?

So me and my boyfriend are having an argument right now because he just told me that he took off the condom during sex without me knowing. My belief is that what he did was rape because he knowingly decided to have sex with me in a way that he did not have my consent. His argument is that my "consent" argument is flawed and that since he is telling me now that everything is okay. Also, he brought up the fact that we had had sex without a condom before that once, but he knew that I had said that I never wanted to do that again after having a pregnancy scare at 17. Let's not forget the fact that since he did not indicate in anyway that he was removing the condom, he did not give me a chance to say yes or no this second time around. Thoughts?