Milk fat % and endometriosis?

Brandy • 29 yr old SAHW. Endo and 1 ovary; DH had a VR. After 4 failed IUIs, currently Pregnant from 1st ICSI IVF cycle!!!
So one of the old hump day health hints talks about skim milk and infertility. I posted a question there but never got a response. Basically, I have heard that dairy aggravates endometriosis. Does milk fat percentage have a part to play in that? For a long time I have drunk skim or 1% milk. I recently bumped it up to 2% milk. Now that I have reached my goal weight and I am no longer restricted to 1200 cal a day, I purchased whole milk. I've heard that whole milk is best for fertility. Or at least that skim milk is bad for fertility. Has anyone noticed more endo symptoms with whole milk versus skim milk? I have felt pain in my ovarian area on both sides since before my period started this cycle, and it's been everyday. My milk intake has gone up in that time. I try to have some everyday.
Basically I am trying to find out if the risks of inflammation and growing endo outweigh the fertility benefits of whole milk. It all seems like hearsay. If anyone has personal experience or reputable scientific articles, I'd appreciate it!
Speaking of lack of scientific evidence, I ate fresh pineapple during implantation time last cycle. Bfn.