I have a nipple biter on my hands

My son turned 4 months on December 26th, he’s EBF. The past 3 days, he’s decided he likes to bite on my nipple when he’s nursing. I tell you the first time he did It I jumped and I freaked out, he cried cause I scared him.... same thing happened with my first baby and that was enough for her not to do it again... my son on the other hand now thinks it’s hilarious and now he’s clamping down and pulling it! I can handle a nip or two but now it’s just plain hurting me. I know it’s probably a phase but should I not react? Or react? I took my boob away from him this evening and he had a melt down for a minute then I gave it back and he fell asleep 🤷🏼‍♀️