Opinions Needed

So not too long ago I started talking to one of my co-workers(which at first, I was very hesitant about for obvious reasons). We've been hanging out and getting to know each other for a couple weeks now. We have a very good romantic connection and just recently (less than a week ago) had sex. We rarely work shifts together and even then we're not working in the same area. Well yesterday, we had a night shift together, of course I was flirting with him whenever I was able to in a playful way. Come time to go home, he seems upset and distant with me so I ask him what's wrong and he says that I'm too "touchy-flirty" with other guys at work and that he can't work with me. It really pissed me off because I don't recall myself flirting, even though I have a very friendly attitude and have been told by people at work that I am very attractive. The certain "flirty" situations(like play fighting, touching someone's hair and apparently I slapped someone's butt with a hanger) he brought up were also at least a week ago but he's just now saying something. Mind you, we are working late on a thrusday night in retail so the store is completely dead so there's nothing else to do other than mess with each other, this happens very often because thats just the environment of the store. I guess my question is should I apologize and stop being flirty? Or is it too soon for him to be acting this way? I also know I wouldn't want him flirting with girls in front of me but I don't find what I was doing as flirty. Mind you, I just got out of a serious and manipulative relationship so I don't want to be told how to act just yet or have to tone down my "flirtyness". But I do really like the guy a lot and don't want to mess this up. What do I do and how do I respond to him?