Lube when TTC

Ok so please be nice.. but I know your not supposed to use lube when TTC because it can slow down the sperm. If my husband finishes inside me and then after we've both cleaned up he then finishes me off with his fingers and some lube, is that bad? I'm thinking it's not because the sperm should already be on its way to the destination, but just want to make sure! Thanks ladies