Why didn't he cum?

My boyfriend of 3 years and I just broke up. Not because of what I'm about to say, but I need to know if this was normal.

When we would have sex my bf would always pull out super early and never cum. Keep in mind we always use condoms and I'm on birth control. I don't know if this is normal because I lost my virginity to him and I've never had sex with anyone else. It's bothering me because even if the guy doesn't do it inside you I thought the whole reason for them doing it was to get pleasure. I feel like I've been giving my all to him and he hadn't been giving it back? I even brought it up to him a couple of times and he said it wasn't a big deal. Is it???