Should I leave him?

Okay, so my bf and I have been together for 3.5 months. Not very long yet we’ve already been through some tough shit. First off, our age difference is HUGE which caused a lot of problems not within the relationship, but how people perceived us and it was really hard. After that problem, I thought I was pregnant by him turns out my period was late because of stress and he later told me that he has had a vasectomy and didn’t wanna tell me out of fear of me leaving him (that’s a huge fucking thing to keep from someone). Next, I find out he has a crazy ass ex that he was with for 15 years, and she cheated on him, he told me he still has feelings for her (not the get back together kind but the anger from what she did) I ignored it because I understood that, but he started talking about her all the time I’m talking days in a row multiple times a day showing me where she cheated and what the guys car looked like that she cheated with, etc. which became another huge problem and after I confronted him about it he stopped for a while. Just recently I’ve been feeling quite distant from him, he proposed to me a couple weeks ago and I said yes, but I’m beginning to think I shouldn’t of because we’ve only been seeing each other for not even 4 months. I went through his phone today which is completely not like me at all. I saw a text from him to his ex saying that he still cared and worries about her (the text was from late July) and now it’s September. They haven’t talked since July, but it’s still weighing quite heavily on my mind. Last but not least, he has kids and I don’t, so I’m not his number 1 which I understand and I want his kids to be his first, but I thought I could handle it much better than I have been. It’s hard knowing that we are never gonna have a first child together, or buy a first house together, etc. we are in fairly opposite ends of our lives and I don’t know what to do, I love him a lot and we laugh and have great times together, but everything I just mentioned has been in my mind for like a week now and I’m just really confused. I need advice big time 😔😔