Dairy & fertility

What do you know about dairy and fertility?  I have a serving of full fat organic greek yogurt almost daily for lunch - it's so good!  I've read that so long as the dairy is full fat that it's ok to eat while TTC...but I've more recently found some info online that says it's best to cut out dairy (full fat and otherwise) all together.  What's the benefit?  What have you heard/read/discussed with your dr?  Thanks!