Am i the asshole?

So i left an 8 year abusive relationship a year ago, 5 months later i meet a guy and end up dating him. He moved in with me and now I'm feeling like its not the right choice. I was so young and missed out on so much because of getting married and i dont know that I'm ready to settle down down again. I feel awful and selfish to feel like this. Between us we have 4 kids. Him 3 me 1. His 2 younger kids mom isnt in their life so over the last several months i they have gotten attached to me and me to them. The baby really thinks I'm his mom because he was sooo young when she left. I feel like the shittiest person for thinking it was what I wanted and now I dont. I'm not happy, and whether he realizes hes doing it or not the boyfriend does so many things that make me feel like I'm falling right back in the cycle of where i was before. I spent every ounce of energy i had finding myself after my divorce. I felt like i had no idea who i was aside from mom and wife. I'm beginning to feel that way again. Hes a million times better than my ex in every other way but i suddenly feel like im being gaslighted about everything. The offhanded comments to make me feel bad about wanting to do something with friends after telling me to my face hes ok with me doing that feels like hes trying to control me with my own emotions.... am i selfish for wanting to go out and have fun and spend time with my friends even if i cant convince him to go? Does it make me an awful person to put myself in those kids lifes and then push them away just like every one else has done their whole lives? I dont know what to do....