Sloppy back rub

I'm always rubbing on SO. Sit on him and massage with lotion sometimes stand up to massage him because I love touching him and making him feel relaxed. I rarely ask him for a massage but when I do he does it with one hand and kinda slap my back as if he doesn't want to do it. It makes me feel like I'm not important or he doesn't care for me. He just never randomly touch on me. My love language (if you've read the book) is physical touch. I just never get it! How can I fix it?