Do you believe in dairy intolerance for breastfed babies?

My newborn - 3 weeks old tomorrow - would pull his knees to his check and scream after every time I gave him breast milk.. The doctor told me nothing I ate or drink would effect him through the milk.

I haven’t had dairy in the last 24 hours and I just pumped and gave him it and he seems perfectly fine. It’s been almost an hour. Should I cut dairy put completely?