Slightly underactive thyroid, ttc for nearly 3 years, ovulating and periods regular

Hi all, I would love to hear from anybody in similar situation. We have been ttc for nearly three years. We have been on public system, had some investigations. Semen test came back fine, hycosy showed tubes open. My GP said my thyroid is slightly underactive ( TSH 4.6) and recommended low dose of levothyroxine. I wasn't keen to take it as I feel fine, regular periods, ovulate fine. Just wondering did getting your thyroid TSH level lower help anybody without any hypothyroid symptoms concieve?? Best wishes to all on their TTC journeys, I'm starting to get worn out from it now, easier sometimes to stick my head in sand and forget all about it.