Fertility medications not appropriate for someone who ovulates already?

Angie
So...I'm confused and definitely want your thoughts. I have a new RE who is known to be the best in the State with amazing success rates. I just had my first visit and he asked me why my doctors gave me infertility medications including Clomid and Letrozole. I didn't really have a great answer other than I am 38 years old and they were hoping to increase my chances of getting pregnant. He said that taking these medications more than likely decreased my fertility and hurt my chances of becoming pregnant. Now I have had time to take in all the information that he said, and it was a lot, and I haven't been able to get this out of my mind. I should have asked him why he said that taking these medications is bad for someone who always ovulates with a total regular cycle. I just want to know how it hurts my fertility and pregnancy chances?? I totally trust him and think he is right because I had more side effects and problems when taking these drugs. I just want specifics about why and how these drugs are not appropriate for me or women who ovulate already and have no infertility problems that they can find. Please help! Thanks :-) baby dust to everyone and happy Easter!!