diet and breastfeeding

Ok so I started eating healthy not extremely bad ''diet'' I mostly pump and I've noticed that now that I'm eating healthy there is nearly any fat in my milk compared to before when I was just eating everything 
Could this affect my baby gaining weight or the nutrition he's getting? I'm still producing the same amount