Husband rejects my advances

Any time I try to initiate my husband rejects me. But not just rejects me he will just walk away, change the subject, completely ignore that I'm trying to be intimate. It's very hurtful. I wish he would just say sorry I'm not in the mood or something but last night we were watching TV and he wasn't really too into what we had on so I was rubbing up on him, sticking my hand across his lap , etc and he shut the TV off got up and said he was going to bed. I asked if I upset him he said no and he just went to sleep. Sometimes I will suggest sex and he will act like I didn't say anything and just start talking about something else.

I've brought it up to him but he doesn't like talking about it but it's making me feel like I'm doing something wrong. And if I don't initiate sex he only tries to have sex a few times a year.

Any advice?