Extended use of birth control

So I am 22 (just turned 22 in Jan.). I have been using birth control since about 14. I started on the pill for about 2 years then I was on the Nuva ring for about 2 1/2. I currently have an IUD (skyla). I have had it about 2 1/2 years now. I want to get it taken out but I want to know if anyone has had any issues getting off of birth control? I do not have any kids, but I plan to within the next few years, and I want to give my body a break from the hormones. So has anyone had trouble conceiving after getting their SkYla removed? Or had a difficult time transitioning to not using birth control after using it for so long?