When is it time to talk divorce

We're on our 4th child and I just feel like I can't go on. He literally drives me crazy 24/7. Stubborn, inconsistent, inconsiderate (especially with freedom & time) I told him I'm scared to have another child and be unhappy with him cause we still argue over the same stuff. As if we haven't been here before. Asking him to go anywhere is like pulling hairs. Don't get me wrong we had good times and sometimes he'll help out a lot but it seems like he only dose that when he wants to leave or go with his friends. He surrounded by people who don't live with their children or have families. Sometimes it feels like he just falls short and it's only because he's careless. He doesn't care if we celebrate special occasions in our relationship, Mother's Day or Father's Day. Baby number four makes me realize how unhappy he makes me as a wife and mother.