Stopping breastfeeding advice?

I need some support and advice. I have been breastfeeding since my son was born. He is 5 weeks now. The only time we did formula was supplementing at the beginning in the hospital because he had health concerns then we stopped when he got healthy. I only pump when I know I can’t be near him. Well he just got diagnosed with a milk allergy. I have gone on an intense no dairy at all diet since we found out on Monday. I am used to a lot of dairy and am a super picky eater so this has been rough to say the least. But the worst part is the anxiety. Every time I eat I panic I missed something or there are hidden ingredients or I messed up and am hurting him more. My dad brought me lunch that was totally dairy free from a restaurant and the process it took to find the food much less eat it was horrible. I just want to cry and my heart is racing when I even think about eating. I also have felt super sick since I started the diet. I am also getting many opinions in all directions and it’s making me feel like a horrible and guilty mom. I know breastfeeding is best but at this point I am considering just switching to formula so I know I am giving him safe food. But I feel like everyone says I am wrong in that, formula is so expensive, and I know there are benefits. But realistically I don’t think I can keep this up long term until he is a year which was my original plan. I am so anxious and in tears for feeling like I’m hurting him now or will be hurting him in the future by not breastfeeding.

Any advice or encouragement?