Teeth problems

So I got my braces off 2 years ago and stopped wearing my retainers about 3 months ago. I noticed my teeth shift slightly. I tried to put my retainers on and it hurt sooo bad to put them on, they fit after a couple hours with still a lot of pain. Will this damage my teeth by forcing them to shift with retainers?