How do I know it's time to give up

So I've been married for 6 yrs and we been together for 12 yrs. And I feel like I dont know the love of my life anymore 😔 we had such big dreams of having a family, a house, and all that good stuff. But I feel like I wont get any of that with him because he refuses to work for it. He makes me feel like I cant ever do anything right. We argue so much lately over the pettiest things. Like theres no understanding between us. Also ttc is a nightmare. I took all types of tests and came back everything on my end is normal. He refuses to check himself 🤦🏻‍♀️. Its just all so frustrating, and I'm just not happy. I feel in my heart it's time to let go but hurt because like I said before hes the love of my life. We been there for eachother through so much 😔. How do I know it's ok to let go.