Please Ladies .. Need advice

My husband bought condoms knowing we don't use them but says he purchased them just in case. Well I notice 2 were missing and he claims he used them to masturbate. I feel in my heart it's not true. He has never done that. I feel like he thinks I'm stupid. I feel betrayed and we've already have trust issues because of his dishonesty in the past. This happened 4 days ago and I was ignoring but I can't. I can't even hug him back and all I think about is him being with someone else. I can't believe it!!! I told him today that due to his past infidelity that I can't deal with it and don't want to continue with the marriage. Am I wrong?