Nursing moms and babies with allergies

My son has a confirmed peanut allergy. He had a reaction to peanut butter and we were referred to an allergist who confirmed. I’m sure if it was an issue the allergist would’ve told me to avoid peanut products but for some reason I just thought of it today. Of course on a Sunday. Does anyone know if I should avoiding peanuts and peanut products ? I ate something today with peanuts in it without even thinking and then breastfed my son about 4 hours later. I’m going to call tomorrow and ask the allergist what the guidelines on that is!