How to deal with family and your newborn

So... I'm trying to handle family in a nice way when it comes to my baby. He was born 9/11 and I'm having trouble figuring out the nice way to tell people to back off. We were in the hospital for 5 days and when we finally got to come home we were bombarded with his family wanting to come over and stay for half the day. I know they wanna see their grandson but I had just had an emergency C-section and adjusting to being a mom. I had a very mild case of baby blues because the first night home I only got to hold my baby when he needed to eat. And I was in pain from just having my baby. Whenever we are with his family I only get to hold him when I feed him and they prolong the time we are with them by delaying cooking dinner. They got a little mad when I told them I wasn't doing well the day after we came home from the hospital and it was because I wasnt able to hold my baby, because as soon as they left and I was able to just hold him and snuggle the anxiety went away. They were also mad when my fiance suggested I stay home next weekend while he goes up to our trailer to do some winterizing with them so I could spend time with my family. It's my parents first grandkid too. Oh and his mom also calls my son her baby. 😐😣