"Sex ain't really my thing"

So I've been with my guy 6 months now and we are serious. But one thing keeps niggling at the back of my head. 

So a few months ago, we had a conversation about sex. He basically asked me why I didn't initiate sex, and was worried that I didn't find him attractive, which was not the case at all. I was just still really shy about it. 

Recently, I've been starting. And the last couple of times I've put quite a bit of effort in (E.G., dimmed the lights, lit candles and brought my blankets and pillows downstairs onto the floor and dressed up all sexy for him as a surprise). But he never even seems to try to start it anymore, ever since I became more confident. 

Yesterday he told me "sex ain't really my thing". Kinda confused on what exactly he could mean by that? Before, he always seemed to want sex. He was always trying to tease me while out on dates or just in public, or we'd go to bed and within a couple of minutes his hands would wander.  But now he seems to have little to no interest anymore. Kinda sucks, I try not to think about it but it's kinda making me feel a bit gross. 

Anyone got any advice for me? I want to talk to him about how I feel in this situation but I don't want him to think that I only want sex, I absolutely adore him.