Why does it only hurt with him?

So I'm in college and as single as I am, I sort of arranged a fuck buddy or as I like to call him, my emotional support "animal" 😏. Well I had made an increasingly dangerous decision to stop wearing condoms, but only with him and this is because it not only felt better without condoms but it just hurt whenever we had sex with them. 😓 The weird thing is it ONLY hurt with him... he is the only guy that I have ever had unprotected sex with and it's only because it hurt so badly. Does anyone know why this would be?