Flange size help

I use the blue Spectra pump and both the 24mm and 28mm flanges seem to not fit right. My nipple and part of my areola get sucked in very far and rub against the side. Does this mean the flanges are too small or too big!?! Or do I just have “elastic nipples” and this will happen no matter what size flange I use?