Sister not invited

We are having my daughters first birthday and baptism party together, it's coming up in two weeks. My boyfriend and sister and her husband don't get along AT ALL! He doesn't like to see them. He said I can't invite them to the party and if they come he'll tell them to leave. I'm not sure if I should just invite them? If I don't my family will be upset with me and if I do then my boyfriend will be upset with me. They live with my parents so if I don't invite them and they still come anyways he said he'll still tell them to leave.