Fantasies About Other Women

So my boyfriend made a comment or joke, whatever you wanna call it last night about picturing the girl on tv in his head while we made love. It turned me completely off and we didn't end up doing anything. I confronted him about it earlier and he said jokingly that next time it's going to be the Kardashians and whoever else. I know they are actors but is that right for him to say that even jokingly because it made my self esteem go pretty low that he has to think of someone else while we're making love..