Breaking up with SO

When did you know it was time to break up with your ex? 
I don't know what to do. I'm engaged to my SO and we live together but he's just being so different lately he ignores me, he won't say I love you unless I say it first and if someone makes him upset (babymother, boss, etc) he'll take it out on me (not physically). I just don't feel like I'm appreciated in our relationship eventhough I've done so much for him. Yesterday was the breaking point for me. We only have one car and I let him drive it mostly because he has children so if something were to happen with them I want him to beable to have the car. He give me rides anywhere I need to go and I needed to go to the mall to get some stuff for our vacation coming up. So I told him to pick me up at a certain time and he never came he didn't pick up his phone, I called his mother to see if she can get in contact with him and he wasn't answering for her either. I waited for him for three hours I texted him asking if everything was ok I called him atleast 20 times and nothing. I  finally called my mom phone asking her if she could come get me and I stayed at my parents that night. He didn't even bother calling me all he did was text me late at night "sorry I fell asleep." I don't know he doesn't seem like he wants to be with me anymore and I just don't see this working out. Any advice would be great.