Who's to blame?

Long story short I was in a relationship for 6 years. A year into the relationship I got pregnant and decided to have an abortion. He supported me and went to planned parenthood together. Now 5 years later (after being engaged for two years and about to buy a home) he ended things. We were trying to get pregnant for the last year and no luck, I definitely think that had an effect on him. He has been bottling up these feelings all these years and I had no clue. I blame myself because I think "what if". He says he doesn't blame me or hate me, he hates that it happened. I suggested counseling and he didn't want to do that. It's been almost two months and I'm still so confused and I feel like he gave up. He says he didn't give up because he tried for 5 years and if he wanted to give up he would have ended it right after the abortion. Idk what to think anymore.