Family visiting after baby 😩

Is anyone else nervous about this? I guess I'm not nervous, I just want to have our time with our Ella. I want to breastfeed at ease and not hide in my room and then have to pass my baby off to my mother inlaw or father inlaw, etc. I don't so much mind my family which sounds horrible but my family is not over bearing at all. My husband's family is such a baby hog. Even taking the baby away from the mother. I know this only because they did it to my sister inlaw. Which I don't think I can handle. We live next door to them so I do not want them dropping by after work every day. I just don't know how to tell them. They cook supper every Monday and expect all of their kids to be there and they get pissed if we don't come. I told her last week we will not be attending anymore and she still calls and says it's ready. 😐😑 just needs some advice on how to handle this! I would never keep their granddaughter from them I just want them to know we are the parents.