When do u know when to let go?

How do you know when it’s time to let go? My bf and I have been in a relationship for 3 years we have had our ups and downs. But lately things just haven’t been working for us. We argue often and I feel like I can never express my feeling with my bf because he thinks I’m just trying to argue. A few months ago we expressed to each other that we weren’t the happiest but we didn’t want to let go so we decided to try and fix things. Fast forward to a few months and the “fix” only worked for a month or 2 and I began to notice that he wasn’t happy nor was I. Last night my bf and I had an argument and he expressed that he thinks “we are just not going to work out” he said it in the heat of the moment but I know he meant it. He claims he didn’t after the argument and says he really wants to make us work and that he just said it in the heat of the moment. I’m not going to lie I have thought that too. It saddens me the thought of losing him. We been thru SO MUCH together thru or lowest points in our life to the highest points. We are each other’s best friends. I know that he has major love for me but I can’t shake but feel the “end” is coming. I know that if we break up it’s going to crush us both. But if I’m being honest sometimes I think breaking up is the best thing to do. But then I also think that we can work out. Idk what to think. Any advice?