Working out affecting fertility?

Kristyn • Married to a very entertaining guy! We have a puppy named Dolce and a baby due in jan
So how much working out is too much? I've been an athlete my whole life and take pride in being physically fit. I lift weights, do cardio, and Bikram yoga. My hubby says I work out too hard -- he barely works out at all so I say he has no idea what "too hard" could be.. But how much working out would negatively impact my fertility?