I can’t be loyal if we’re not in a relationship

I’m confused as hell this guy I’ve been casually dating expects me to not date other guys or have sex with other guys. That’s stupid because we’re not committed or exclusive. He says I’m not loyal I’m not dumb he just wants me not to look around. I’m very big on not keeping my eggs in one basket. Because I don’t know if he’s “loyal” to me. Any advice?