2 week old super fussy at breast

Ca
He's feeding ok, but sometimes he gets really mad and grunts and starts screaming and pushes away from the breast like he's done, but he still has my nipple in his mouth. He'll stop crying after a minute and keep eating, but I'm not sure what's wrong. Anyone else have this problem?