Worried

I got my IUD removed about 2 months ago before my husband deployed. We figured since we wouldn’t be having sex this would be a great time to get it removed and see if I could have a regular period. I’m super worried about my fertility because my mom had a lot of miscarriages and I have PCOS. I was told that my PCOS is mild and that I have a low chance of having fertility issues. I know periods are supposed to be between 21-35 days apart for ovulation to occur. I counted today and my most recent period was 4 days early—under 21 days from my last one. Has anyone had IUD removed and had their periods take a couple months to even out? My doctor said to let her know my periods were too far apart or too close together so we could talk about options but I’m hoping it will fix itself. I don’t want to tell my husband that there might be an issue until I know I’m really having a problem. We wanted to start trying when he gets back and the thought that it could be an issue makes me feel sick.