Allergies

My baby has A cows milk protein allergy so I have gone dairy free and switched to oat milk. He just turned 6 months and He had a reaction to oats as well ..my doctor told me to obviously not given him anymore oatmeal / oats . I didn’t think to ask if me drinking oat milk is still ok. Would that cross over into breastmilk?