Wrong flange size?

So I started off using 24mm flange sizes. They fit fine at the beginning of my pump session but then my nipple swells and rubs against the sides of the flange. Is this a sign I should get a bigger flange size? Or is this normal?