Family always wants to come over and see my baby but I’m exhausted

Is it normal to not want to have visitors for a while after birth? I have a newborn baby and my mom always wants to come over. Sometimes I say yes and she used to help by cleaning or bringing food. But now it’s just to hold my baby while I clean and cook. And she tells me how much I should take a nap or not think about anyone else during this time but then will go on on some story about her new dog or drama with work. It’s so annoying I don’t want to hear any of it. I just want to hold my baby and bond with him. My sister also wants to come over for lunch but I don’t have the energy to pretend to be interested in all their drama and stories. I feel I don’t have time for a social visit because I need to nap during the day while my baby is napping bc at night my baby hardly sleeps. Usually I see my family once/week but they want to come over every weekend and during the week too and it’s too much for me.