High cholesterol and TTC

I read an article this morning that said if you or your DH have high cholesterol then you could have difficulty TTC. It does make sense. DH and I have both been tested by a fertility doc and everything came back normal. We both have high cholesterol (neither is overweight but a strong family history) and we have been trying for almost a year now :( anyone have the same issue? My levels are all amazing except for my overall cholesterol because of the history.