Cow's milk protein allergy/intolerance

Anyone with newborn diagnosed with CMP allergy? My 8 week old LO was recently diagnosed with cow's milk protein intolerance. Pediatrician advised me to have non-dairy diet. Do I have to totally cut off everything dairy or a little but here and there is fine? I find it difgicult to do so since I'm a big milk drinker, and totally love butter and cheese. It's a little frustrating but I'm trying for my LO's sake. Anyone experincing this?