Should I go to the doctor??

A

So I have noticed for the past year or so that I have a dry yellow substance in my nipple holes. It has a taffy like texture when I remove it and it is very painful during removal. After removal it makes my nipples hurt for days afterward and then it comes back. I have never been pregnant nor am I pregnant right now so I am a bit confused as to what is causing this to happen? Is this normal?