Dairy allergy, switch to formula?

I feel like it's been an uphill battle when it comes to getting my baby breast milk. I can't breastfeed because my nipples are inverted and he never did latch right. So I've been exclusively pumping the last 8 weeks. Around two weeks I noticed that he was getting fussy after feedings and spitting up whether it was breastmilk or formula so we suspected a dairy sensitivity. We took him off of formula all together and I quit drinking milk. That seemed to do the trick. I saw a big drop in supply around four weeks, and we had to supplement with a little soy formula. I did absolutely everything I could and eventually I got my Supply back up and then some. I finally have a freezer stash of just over 40 ounces and today my baby had blood in his stool. They're suspecting a dairy allergy. So this means if I want to continue to exclusively pump for him then I have to cut out all Dairy. This also means that my current freezer stash is no longer any good for him. I feel so defeated at this point I kind of just want to throw my hands up and give him formula. I don't know what to do it's going to be extremely hard for me to cut out Dairy I feel selfish if I don't.