Anal sex

My partner has mentioned a couple of times about trying Anal sex. Might sound stupid but I'm actually scared to try it. When we are having sex he keeps trying to put a finger in, always stops if I say no but then I feel awful for saying no in the first place. Does it hurt? Can anyone help? Thanks :)