Going down on me after finishing inside

Ok, I'm confused about guys aversion to having their cum anywhere near their mouth. The guy I'm seeing won't even kiss me after I've swallowed. I know what I taste like, it's not that big a deal. And it's not like I have a fetish around him tasting his cum, but I'm somewhat insulted/annoyed that it's perfectly fine for me to give him a bj after he's been inside of me, but not ok for him to finish me off orally after sex. Any ideas? Any suggestions on changing his mind, or at least any explanation? If it's a hard boundary, I'll leave it, but I feel like this whole thing is tied up in toxic masculinity and the asinine fear of being perceived as even the tiniest bit gay.