Sneaking around with my ex

So me and my ex reconnected and lately and have been messing around with each other and now are going on dates. I can’t tell anyone because my family and my some of my friends don’t like him. They feel if I go back to him I’m stupid. I agree that we didn’t have the best relationship. When we first got together everything happened so fast. I got pregnant after a month of dating and then miscarried. Then we got engaged but began fighting a lot. Then there was cheating (on his behalf) and more fighting. We were on and off for 3 years. So we have a lot of history, which is what draws us together but also what breaks us up. We did a lot of petty things to one another until I got fed up and walked away. We would reconnect and be friendly but when both of us started dating other people the other would get upset and we would stop talking. So we went 2 years of not dealing with each other but of course as time has passed we began missing one another. This time we are both single so we’ve been having fun and trying to be better people with each other. I know there’s a big chance this could blow up in my face but the way I feel when I’m with him is unexplainable. He is the only man I’ve ever loved and even though he wasn’t the best everyone knew how much he loved me. Lately I’ve been saying I’m out with my girlfriends instead of with him cuz I just don’t want to deal with any negativity. He’s been doing the same since his friends don’t like me either. Am I crazy for wanting him still? I’m feeling confused