Newborn and visitors?

I'm a FTM and I just had my baby on 8/20. So she is almost 10 days old. Now of course everyone is so eager to meet and spend time with the new baby. My family is farther spread out and have greater distance between us. However DH family is all within an hour drive from us. But there are those who haven't gotten a chance to meet her but those are more distant relatives my SILs kids (8 and 3)and her husbands mom, etc. And there is a birthday coming up on Saturday for one of the kids. So for some reason my DH thinks it's important for us and our newborn to go to the gathering so everyone can meet her. However I'm skeptical and dont want to bring my newborn daughter to be passed around and held by several people. He thinks and says she'll be okay but I'm not ready to do all that. I'm unaware of who all will be there and dont feel like being in that position with my DD. I told him they'll get a chance to meet her when shes older and it doesnt need to be on the birthday. Hes still upset about it and it makes me upset because his family isnt very considerate of how others feel. And I think he told them we arent going just because I don't want to. Which I feel makes me targeted nd look bad. Am I doing the right thing saying no to going??