How to tell family no spanking?

I've re-written this about six times because I just don't know how to ask for help on this.

Long story short, my husband and I don't spank my toddler and we don't plan to (please do not try and persuade me to spank my kids, my mind is not open to the subject of spanking). We also don't want anyone else spanking him. I think this is a reasonable parenting rule to ask family to respect, but my husband's family is giving us a hard time about it. Specifically my FIL. I caught him whispering (loudly) to my husband the other night that we just needed to start spanking my toddler or he'd turn into a brat, and my FIL constantly threatens to "beat" my child. I'm done, I've had 3 serious talks with him and my husband has had 2.

I don't know what else to do? I have quietly made sure that FIL and other select family members aren't left alone with my child for his safety, but I feel like I need to do something to make this talk stop. I've tried to be as kind as possible, but I've lost my patience. Any advice?

I'm trying to be as patient as possible, because I know these family members were raised in a different time. Also, we don't just let our toddler go crazy, we do other parenting stradigys like taking toys, time outs, etc.. for rude behavior.

*Update* Thanks for all of your input. I knew that possibly withholding visitation probably was the answer. I hate for it to come to that, because my son loves my in-laws. Also because my MIL has been very supportive of our decision, so withholding visitation from FIL would also mean MIL wouldn't get to see him. This is going to be a hard conversation to have, but it's looking like my last option.