Need advice on BF's wandering fingers 😒

Almost every time we have sex, my boyfriend's hands and fingers get dangerously close to my "other door." I don't know if it's intentional but it's consistent. Do I ask him about it? I'm not interested in trying anal at all.