insecure as hell

Me and my boyfriend...... Ok so long story short, we met last year, and where dating for 6 months. I got pregnant and didn't tell him for the entire 9 months. Because I found out he had a gf and two sons. When we did get talking things over he said he isn't in love with her, hasn't been for years, and that he never did want to be in a relationship with her, but she got pregnant right away and he stuck by her but she got pregnant again straight after there first. So for the last 6 years he felt trapped by her. So once it all came out about me and the baby, they broke up. And over time me and him started to build on things. I'm insecure tho that it's too soon after his break up. And that maybe he will go back to her. He has tried and tried and tried to keep reassuring me that for him there is no time needed. Yes for her there is, but if it will prove to me how much he wants ME, then he will tell her. I don't want him doing that as the girl has been begging him back. But I still keep being insecure that maybe he will change his mind an go back. He has said he isn't at all attracted to her in that way. And isn't in love with her. But how do I know. I can't shake the constant thoughts of paranoia and mistrust. 😕😔😔😔😔....I really want to continue as we are and see how we go, but at the same time I'm scared and thinking am I best to back off and leave him for a while to see what he really wants. He keeps telling me he already knows what he wants and doesn't need time. And that's me. But I don't know 😔