Birth control too long

Brigid
I had the IUD for about 4 1/2 years and got it removed to start trying to get pregnant. Does anyone out there know if the longer you tell your body to not get pregnant the harder it is to get pregnant? We have been trying for two months so I am trying not to get frustrated I just want to know all the information I can get in this endeavor.