Bf masturbation material: Past sexual experiences

I made the mistake of asking my bf what he thinks about when he masturbates, he answered honestly and without hesitation: women that he's had sex with. I couldn't believe my ears. I'm shocked, hurt and confused all at the same time. I really don't mind if it were abstract fantasies of strangers or even porn, but recreating mental imagery of past experiences? He says that all men do that and I'm just being naive. Is this a red flag? (Side note: he doesn't watch porn b/c he's a recovering addict). Anyway, I can't help but feeling insecure that our sex life is just not cutting it for him. And in the interest of full disclosure, i masturbate, but I think of scenes that we've done or I watch lesbian porn.