Physical Activity Preventing Pregnancy

I've heard being overly active can prevent conception is that true? I do CrossFit 7 days a week because I love it. But some people have told me that being that physically active can be detrimental to conceiving. Any truth to this? Do I need to cut back on the exercise in order to "get it to stick"?

Thanks :) 🏋️