Should I teach my children ASL before I teach him to talk?

I am hard of hearing with a degree in ASL. I love the language, community, and culture. My beau has come to like it to (he was a little taken aback but he understands it now) so when it comes to teaching our kids ASL, I want them to be somewhat fluent. BUT every parent and family member keeps telling me it’ll make it harder for them to talk. I know ASL is best learned when in infancy and beyond, but now I’m afraid of the backlash that happen if my kids sign but can’t speak.

side note: if they’re Deaf then the speaking part isn’t a big deal. I’m just afraid that my hearing children will be made fun of if they don’t speak at the same rate as others.