Why do I feel so guilty?

My boyfriend and I adore each others company, however, when he is with me and my parents end up being in the same room as us and we’re having a conversation, my boyfriend will start rubbing my back and almost like stroke me. It pisses me off and it also makes me feel uncomfortable as my grandad used to do it to me when he was drunk to try and act like everything was ok ans my grandad used to be an AGRESSIVE alcoholic. I told my boyfriend that I don’t like it and explained why. I feel guilty because he thought I liked it and I thought I could get over it but how can I expect the relationship to work if I keep in the things that make me feel uncomfortable and not be honest with him, right?