starting exercise while TTC

Is it a bad idea to start exercising while TTC? I've heard it can stress your body and hinder your attempts. I'm not talking about starting something crazy or hardcore... More like a moderate 20 min on the elliptical three times a week. I'm not overweight and I'm not looking lose a certain amount, im just feeling like the holidays and stuff caught up with me and I'm not loving how my clothes fit. Just want to get into a little more shape!