Pregnant and husband doesn’t want my dad around the baby

So I’m pregnant and my husband doesn’t want him around the baby. My dad isn’t the nicest to me and he will say he doesn’t love me to hurt my mom. He uses me to just really hurt my mom. So my husband says it’s toxic and the baby shouldn’t be around it but to me it still her grandpa no matter how much he hurts me. I really don’t know what to do