How soon should I remove the iud before trying to conceive?

I have the paragard, have had it for 2 years now. My so and I have been talking about trying for a baby soon but I’ve heard that I should take out the iud a few months before we start trying because it can take awhile to conceive after having it removed. But I have heard a lot of people still trying or tried for a long time before getting a positive after having the Mirena. I haven’t heard any stories about the paragard though. Would it be the same thing? Should I take it out soon? For those that have had it and had it removed, how soon did you get pregnant?