Should I give up on my relationship?

We've hit a rough patch in our relationship. At the moment there's no affection, no effort on his part, he's even said he doesn't care about us (during an argument). But sometimes I can actually sense that he doesn't care. I seem to be the only one trying to make this work sometimes. I don't really want our family to break up because when we're good, we are brilliant! I do try and speak to him about our situation at the moment but he hates any kind of serious chat and just clams up, shrugs his shoulders and says whatever. I tries again yesterday and asked him outright what he wanted. He said he wasn't really bothered either way, then I said so you don't care if I walk out of this house and be done completely. He then said he didn't want us to split up 🤷🏻‍♀️ so again that leaves me confused because if he doesn't want us to split up then Where's the fucking effort to keep us together? I don't know wether to keep fighting for us or just back off. I love this man so much which is why my head is so messed up. I'm also on anti depressants (for other reasons as well as my shit life at the moment)