Could I be infertile or just not having sex right? Lolz!

I got my IUD out on 12/29/15 and my fiancé and I have been TTC for baby #1. Before I left my appointment I asked my OBGYN if there's anything I should do before trying because I had read on a baby website that women who've not had children with Mirena removed usually conceive within a month. All he told me was to wait one to two periods before trying. I was like ok and I left. I have a history of an eating disorder. I'm in recovery but still am underweight for my height. I'm 4'11. The reason for that is because I don't drink anymore like I used too. I almost died from alcohol poisoning. If I do drink now, it's only a small glass of wine and even then, I don't finish it which explains the weight loss. Plus I eat healthy. We moved back in with my parents to save up money before we move down to Myrtle Beach in April and I enjoy cooking for my fiancé and I. I grew up eating healthy too and I drink plenty of water. I've started taking pre natal vitamins but is there a chance I could be infertile or are we just not having sex right? My fiancé and I don't smoke or drink and he's healthy too. He's skinny like me so his health is not an issue. What am I wrong? My doctor never told me I couldn't have kids. I'm worried because it took me so long to recover from all of my trauma and I'd hate to go through another round just to be told I can't have children.