It still affects me

It's been 2 years and I'm in a long term committed relationship now. This is the first time I've been in a proper relationship before. It used to just be casual flings where I'd sleep with the guy 2 or 3 times and dump him or just one night stands. I love my boyfriend and he's never done anything to hurt me or make me uncomfortable but sometimes, after maybe a weekend or a week where I feel like we've had a lot of sex, I start to get depressed and angry. I don't want to be around him and I don't want to have sex. But I've never said no with him before. I don't want to punish him for my past or maybe I'm subconsciously scared he won't listen and do it anyway, although I know he wouldn't and he knows my past. It's just difficult to remember that my body I'd mine.