Start infertility workup early?

I’m 31, husband 33, trying for our first and no luck after 3 attempts. Neither of us have any medical conditions, eat healthy, don’t smoke, and exercise regularly, not too intense, and I’m average weight. I have regular cycles between 28/30 days and stopped Mirena in August after having it 6 years. I know the guidelines say wait until you’ve tried 12 months before going to have an infertility workup if you’re under 35 years old, but I can’t imagine waiting that long and every month we fail brings me closer to thinking there is something wrong. I’m using OPKs and CM monitoring and timing BDing around peak days well. It seems like we’re “doing everything right” which makes me think something is wrong.

Did anyone go see their doctor earlier and start infertility testing? After how many cycles did you try before going? What age were you?

I feel like at least if I knew there is no problem with him or me I could rest easier and accept that it will just take time.

Any and all advice welcomed, really struggling tonight. 😞