Confused?

I was having sex with a man tonight, we've been hooking up sportatically for a few years now. Not serious and there is quite a age difference between us. Anyways we were laying in bed, and I was faced the opposite direction from him, looking away, for a good bit of time, then I randomly faced him, laying on my stomach, and just stared off into space, causally making eye contact a few times. Then he proceeds to say what are you looking at? Taken by surprise I quickly said I'm just looking, he said well don't face the other way, please. 
Is there a specific reasoning for this?