Im tired of all this and feel like ive wasted 3 years of my life

I met my boyfriend 15 years ago but we were just friends at the time well we decided to get in a relationship a little over 3 years ago which was around the time I was in the process of trying to leave my then abusive husband. Long story short on that I broke things off 6 months prior to being able to get away from him. He had no where to go so me having a heart i let him stay but things got way out of hand so i just had to leave my apartment with all my belongings and never look back. Well around the time I left I was talking to my now boyfriend which was my best friend for so long and he tells me if I need some where to stay I could come live with him. I had every intention of waiting for my lease to be up pack all my stuff go move in with my mom but ex pulled a gun on me cause I wouldnt tell him we could be together eventhough he wanted a open lifestyle no kids locked me down i wasnt aloud to have friends go any where without him he never worked or contributed to the household but yet made me hand over my paycheck each week and ask if I could use any money which I only used at work so I could eat... it was an absolute nightmare I just wanted out.. but anyways I couldnt wait another month for my lease to be over so I could leave due to I felt my life was in danger. I ended up going to my now boyfriends house. First 2 weeks were amazing. We had a great relationship and before we got together we clarified what kind of people we were such as how we acted towards our significant others while in a relationship. He was everything I was looking for and then some. About 4 months later I found out I was pregnant which was not planned at the time eventhough we both said we wanted kids. Before even finding out about the pregnancy things had already started to change. All the things he said he was had changed. I felt lonely all the time and he barely had anything to do with me. I contemplated leaving and just getting to know myself with out any relationship but then I found out I was pregnant. Through the whole pregnancy he stayed more like a friend to me than my boyfriend and I stayed and i am still with him today. I'd tell him how I felt atleast once a month only in hope he would try something anything to make me feel we was more than friends but as usual nothing. I ended up having my baby early due to complications with blood pressure but everything turned out fine with the labor and delivery. After having our baby thats when he tells me he never wanted kids eventhough before we got together he said he did. He said he didnt know how he would be as a father and didnt want to be one. We talked through that because what else could you do you was already a father and had a baby it was too late for those kinds of feelings and I just chalked it up to he was nervous that he wouldnt be a good father. 6 months after our baby was born he was helping a friends ex girlfriend/baby momma get her vehicle back from 1 of her ex boyfriends. I am secure in myself and our relationship I had no problem with anything he was doing but then I started noticing he'd walk outside through the yard down the street talking to this girl and instead of maybe and hour or so conversation it turned into 5+ hours then 8+ hours every day while I was inside caring for our baby by myself which he never really did much to help anyways and i never minded cause I like things done my way but come on that long on the phone with someone you dont even know. Well while he was on the phone with her all day one day I texted him and said "do you love me" less than a minute he was off the phone walking back into the house and the proceeds to tell me he dont know. We hadn't ever even had a fight or anything go on through our relationship at this point which we was at our 1 year mark in our relationship but he tells me hes not sure he wants to be with me and how he wants me and the baby to go stay a few weeks with my mom. I said sure but while I'm at it im packing all my shit and taking it with me cause if I have to leave over the fact your not sure you want to be with me and your baby theres no reason for me to hold onto this fake ass relationship. I had almost everything packed and was ready to go. Thats when his mom talked to us mostly him about remembering the love we had when we made our baby so he changed his mind and wanted me to stay I was iffy about it cause i had already been through a lot in life I couldnt do anymore crazy but I stayed. We got through it. Things never changed he never put his arm around me hold my hand kiss me or even acted like i exist. I still mentioned how lonely I was and how I'm sick of having a friendship when he was suppose to be my boyfriend and care for me more than just treating me like a friend. Its been a little over 3 years now its still the fucking same and i am still so fucking sick of it. I mean literally he is so weird about it and I get nothing but conversation from him thats it. I cant even get a simple hold my damn hand touch my damn shoulder something. We hardly ever have sex. I mention that all the time now too. I feel like all i do is bicker at him nag him but why not what else is he trying to change? Nothing and makes no damn effort in showing me he even gives 2 shits for me. When something happens to me and i am upset or cry and he sees me he just ignores me like nothing is going on. No comforting me nothing not even ask me whats wrong. He simply doesnt give a shit. I am tired of being the nagging girlfriend and constantly having to say something and when I do try to talk he just laughs at me like I'm making jokes or something. When we do get to have sex its when ever he wants it which could be if i am lucky once a month. He cums to quickly cause of it being so damn long since the last time we done anything that when hes done he says well i poked it in you if you didnt cum thats not my fault i tried but when he pokes it in me he goes so damn fast just trying to get himself off and then doesnt do or try anything to get me to finish. I cant even tell you the last time ive even had an orgasm and i love sex i enjoy it and want it every day. I am 29 years old I should be having all the sex I can before i just loose the want for it and i cant even get any. Its always the same 2 positions either me ontop or doggy. Thats my options. Cant make love or hold each other nothing. He wants me to go down on him i do. I cant even tell you how many times i have. He has went down on me a total of 3 times ever and only tried to finish me which he only ever stayed down on me for 30 seconds but finished me once out of of the 3 times. Im tired of being lonely sad hurt horny needing more than what i get and putting more effort in our relationship to only get shit on in the end. I love this man which I cant tell you why i just do. He has no intention to ever marry me he made that clear and this is the longest relationship he has ever been in. Its sad after ive wrote all this im like wtf am i thinking staying with someone that makes me this unhappy cause the reality is things will never change so instead of asking what should i do im changing it to how do i break up and end this unhappy relationship on good terms for the sake of our child