Pacifying baby

So I'm exclusively breastfeeding and my little man is a champ at it . But he has started this new thing ... He's pacifying ALL THE TIME for comfort or to fall asleep. He will cry and cry until I let him, I've tried a pacifier .. Which he will normally take but here lately he just wants to have my nipple in his mouth.  I have tried fighting him but he will cry until he gets it .. How do I fix this ?!  Will he grow out of this ?!