Condom or me?

When me and my boyfriend have sex he can never finish with the condom on. He always has to pull out and finish. In the beginning of our relationship we would have sex with no problem with the condom on but now he has to take it off. Should I be concerned that there is that a problem with me?