How old is baby? Whole milk is not nearly as nutritious as what they fortify formula with. I don’t think it would be enough calories and depending on age could be deadly to not give your child adequate nutrition. I’m not a doctor so I can’t give medical advice on this but think of those news stories you see where babies die from not being fed properly, and depending on the age and what else baby may or may not eat, seek a second opinion. I’d get a second opinion anyway if it was me just to be sure you were doing best for baby