Really need some advice

I’ve talked to a few friends, but they just say “yeah” and we move on. But I really need some advice. When my boyfriend and I got together, I was married but separated. So we had a countdown to the day I got divorced (we had to wait a year after separation). This past Wednesday was finally the day. And we got into an argument the night before. He told me I had hurt his feelings when I got home after my divorce was final. I started to cry Bc I felt bad for hurting his feelings. He then told me he needs to rethink if he wants to be with me or not because he feels as though he is hurting me too much. This man wanted to marry me one day. And now all of a sudden this? We live together, created a life together, have so many future plans together. And now it’s like he is second guessing us. I’m trying to stay positive because he told me he did want to spend forever with me and does love me. But I find myself wanting to build a wall back up. One that won’t allow myself to continue growing with him because I am scared he will end it whenever.