Is it harder to get pregnant when you’re very physically active compared to women who aren’t?

Just curious if other very physically fit women have had trouble TTC as quickly as their girlfriends who aren’t in shape? I’ve noticed a trend amongst my friends and family that the women who live a more sedentary lifestyle get pregnant very quick compared to the women who workout everyday.