Not sure what to do anymore

I have been with my SO for 3 years on the 16th. We have an 8 month old together, he calls me his wife and I have a promise ring. For the years we have been together he has "Forgotten," our anniversary, my birthday and Christmas. I have gotten him small gifts for those dates. I am not a material person, to me it is the thought that counts, just saying Happy Anniversary/Birthday is enough. But I feel like he doesn't think about me, he says it's because he didn't celebrate those with either ex wife (2). I have to remind him that I am not his wife but it is the same thing over and over. He talks about getting married one day but I am not sure I want to anymore, I've been married before and I put everything into it just like I do with him but I get nothing in return. We also haven't had any kind of sex since we concived our son. Our conversations are one sided, mostly me talking about any and everything. I've tried to talk about everything that is bothering me but it's like he doesn't get it or just not listening. What do I do, I want to walk away. He barely interacts with our son, it's me all the time, even on his days off.