Opinions: Is the parents' happiness worth it?

So my husband and I have been having issues with intimacy and affection lately. For me, it's normal. We have 3 kids in the house, one is just a baby. I feel as a mom that most of our attention should be focused on doing things for them and making them feel loved. My husband's concerns are that the kids will leave and we'll fall apart.. But his clingy, needy, complaints are just pushing me away. We had a talk about splitting up, and I've thought it about a lot before, but for me, the negatives outweigh anything positive. We bought our house not even two years ago, we planned on raising our family here. When I think about it, I get so mad at him because if we do split, I'll bare the brunt of the kids' anger and anxiety. They'll blame me like all kids of divorce do. And they'll rarely see him.