Partner cold feet at 21 weeks

Hi my wonderful ladies! I need some support and any insight is appreciated. Me and my bf haven't been together that long but after three months we both really wanted to have a baby, and we'll with my age (39) we went for it. We thought it would take over 6 months to conceive and it happened first try. I felt grateful and he has been the best up till now. He has always asked about the baby how I'm feeling and even accompanied me to appts. For the last two weeks he's been distant disconnected and after some talk he expressed that he felt it happened too soon. He also said he hasn't been in a long term relationship for more than 5 months, he doesn't know how to be in a long relationship. He is 8 years younger than me, I told him why wouldn't u tell me this before I would have kept it casual. He said he'll be there for the baby but doesn't know what will happen to us. I am shocked and crushed. I went from my happiest to nonstop crying. We talked about how important it was for me to have our son live in the same household. He didn't have his father growing up he past when he was a baby. I feel betrayed but mostly sad. We talked and he says he isn't going anywhere but I feel the energy is different. Should I let him go? I need to be at peace. He worries so much about me screwing him financially and that is the furthest thing from me head. Advice?