When to leave

How do you know if you need to leave or just try harder in the relationship? My husband and I have been together for 4 years in May and here recently things just haven’t been the same. We fight all the time about the stupidest things. If he has a bad day or someone does something to make him mad he takes it out on me by starting a fight. He doesn’t turn me on anymore and threatens me with divorce when we fight. I use to cry when he would say he wants me to leave. I don’t anymore because it just doesn’t hurt as bad. Do I need to try harder to make our marriage work or do I just need to leave?