Help!!

So I met a man about 3 years ago, I'm in my twenties and he's 40. For the past three years we have been hooking up occasionally, in the beginning it was frequently, then after a while it began to get boring. So as we stopped having sex with each other for about 6 months, we started In again, and let me say I'm addicted. Not to the sex but him. I have strong feelings for him, and I have no idea if even feels the same way. I don't want to ask because I'm afraid he will think differently and I'll lose my friend all together in the end. The other night we were having sex and whe we had got done we were laying in bed I was facing away from him, then I had randomly turned over and faced him. I started to state off into space, not looking at anything specific. He said what are you looking at? And I quickly said just looking, he said well please don't. Why is that? I felt almost insulted.