Does drinking low fat milk interfere in fertility?

Ish

Im 33 and TTC for about 5 months now. I have switched to using skim/low fat milk and switched to decaf to reduce caffeine intake.

I just read that low fat milk will make you less fertile and it’s better to use full fat milk.

Was I doing bad to my body by Intaking skim milk?