Anal sex??!

I'm not really educated in all of this but I would like to learn something. This guy I'm currently seeing wants to have anal sex. I've done it once with my ex and I did not enjoy it and therefore I don't want to do it with the guy I'm with. He keeps asking me why I don't want do to it and I just tell him cause it's gonna hurt me .. and it is! I'm not lying about that part but he doesn't know I've done it already. And my question is anal sex always painful... or what?