Am I justified? Divorce.

I am really considering a divorce. I never, ever thought I would say that. I feel immense guilt and loss even just thinking about it, but I'm not sure I can continue. After four months of dating when we were 16 and 17, I got pregnant. He partied my entire pregnancy and pretty much left me to be alone constantly. We were together, but I felt alone. Everyone pressured us to get married, so he proposed. 10 days before our wedding I saw extremely inappropriate texts between him and a girl named Mackenzie. He apologizes and I am young and naive so we get married on our scheduled date. I have since found womens sunglasses in his car that he cannot explain and a few months ago he sent Mackenzie, the girl he was supposed to have no contact with, pictures of our newborn daughter in the hospital. 3 years later and he apparently still has contact with her. He went out of his way to wish her a happy birthday also, while forgetting my birthday all together. We have gotten in fights where he has told me he wants a divorce and says nasty things but apologizes and after and says he doesn't mean it. It still hurts and I can't forget the horrible things he has said. He's punched holes in the walls and has a horrible temper. He smokes marijuana, which I am completely fine with, except that we cannot get along unless he is high. He is irritable and mean if he is sober. He puts me down and makes me feel bad. He refuses to stop because he says he needs it for his anxiety. He is obviously addicted mentally. I do literally EVERYTHING for him. Get him any drinks he needs, food, make his lunches, get up at 5am even though we have a 4month old and make him breakfast, anything he asks I do. I stay home with the kids and do every bit of housework. He doesn't want me to get a job and holds the fact that it is "his money" over my head. He goes out at least twice a week, which I don't mind, but gets very angry at me if I want to see my friends (once every 2-3 months). He does NOTHING for me. There is no sacrifice. Anytime we go out to eat, we go where he wants to go. Anytime we go out, we do what he wants to do. I feel like he is a third child, but he is spoiled rotten and I am not allowed to tell him no. There are a lot of good moments and I do love him. I don't think we are right for each other. I do not feel that intense love for him that I have felt before with others. I think we really did rush into marriage trying to play the part of "happy family" and now I feel stuck. I have told him soooo many times all of my feelings and given him direct things to do to improve. He agrees and then never follows through. All of his family members are exactly the same so I don't think it will ever change. I don't want to nag him and change him. I shouldn't have to. He is a great dad and he does provide for us. He just doesn't get it. He thinks that since he works he doesn't need to contribute anything else. At all. emotionally, physically...I am just so lost on what to do. We do not have the same interests. This sounds absolutely horrible but I feel like he is under my intelligence and I crave deep conversation and connection. We just do not have it. I don't know if I thought it would come eventually or what but I am regretting the marriage and feeling guilty for wanting to break up our family. I feel disrespected often and I don't know what to do. There are only so many conversations i can have about this topic before I go crazy. I want the best for him and I would like to have him in my life and co parent with him, but I feel like I have lost myself. I am scared of how he will react when I tell him I'm considering divorce. The other day I asked him nicely if he could compliment me more, he said "you should know how I feel. I tell you nice things all the time". He will never admit that he is wrong or needs to improve. I bought little husband and wife books for us to read and fill out together and I was really excited. He laughed and refuses to do it with me. I just don't know what to do...please give me advice. I am feeling so lonely and so sad about this..