Pulled out for a year.

Ashley • Married mama of 2.
So I'm 34 and my husband is 28. Last year I went to the ER with bad bleeding and ended up having my Mirena removed due to an infection. My body never has agreed with birth control, so I decided to take some "time off".  For the last year, we haven't used any birth control, and my husband has just been pulling out.  Well, I saw my midwife today. She asked a lot of questions, and explained to me that she was concerned.  That after a year of unprotected sex, pulling out or not, chances are good that I should have ended up pregnant. 
The thing is, we were actively trying to NOT get pregnant until this month. Is it possible we were just doing a good job? Lol! She said with my age, and the fact that we've been unprotected for a year, that we should only wait for 6 months before seeing a specialist. Do any of you have experience in this department?  I'm letting this news really bum me out.  :(