How to Throw a family party

When we throw a party with family, we also include a couple of people that we just tolerate for the sake of family but hope they won’t make it. However, they always are there just for the food and hang. We don’t converse at all, they come late, leave early and take food and cake with them on top of everything else. But we still remain respectful, welcoming and attentive to all the people we invite. However, the same ones, when they have a birthday or something, they never offer to drink, or to takeaway cake. They can’t be bothered to even get up and say bye! But for others they are escorted to the door. My question here is, how do I kindly tell them no cake to take home after the stay? Is there a graceful way to do so? The cakes are set in buffet style. I’m also thinking to not attend some of the functions they host because they are so rude.