Should this bother me

my husband and I have had bumps in the road ... this marriage hasn't been easy... I've never slept with anyone but him since we've been married . however, when me and him have taken 1-2 weeks off from each other, which wasn't many times.....he has slept with other women. and of course, I never know during our break because when we do talk , he swears he's not talking anyone. I just feel like if I knew the truth and that he was in fact seeing other women rather for casual sex or romance, I would not have went back ...he lied, got me back and then I find out the truth months down the line..... it just really bothers me that he was inside other women.. getting to know them etc... then came back to me...I really want to divorce him because things aren't the same anymore .... mainly for me