Do I only love the sex?

Okay so I’m in a relationship with this guy I’ve known for a long time. We get along really well and I feel that I love him.

Sometimes I get this feeling or this thought in my mind & it’s kinda like... he annoys me so much & I don’t feel very attracted to him. But I love the way we fuck. Like he knows how to be rough but gentle.

Idk is it all in my head? I’m so confused