Too much coffee bad for sperm?

Me and my man are ttc so I've cut back on coffee I actually don't even drink it anymore that's how bad I want a baby. However, my man has not sometimes he drinks three large things of it a day 😣 guess my question is could this be a reason why it's hard for us to conceive?