Opinions? Advice?

So I’m married to a Hispanic, and his side of the family never talks to me. I have two kids. A two year old, and a 10 week old. They never even come see the kids. Yesterday we went to a birthday party (I dreaded it so bad) when we got there all the “family” members came up to me and signaled they wanted my baby. And then just ran off with him. It really upsets me. They don’t come around, and aren’t in my sons’ life. And plus who just runs off with your baby and you don’t even know the person? I kept asking my husband to get our baby, and he’s like, “it’s fine, we’re about to leave.” It made me so mad. I’ll never go back around. They could’ve at least talk to me too. I was just the loner at the party it felt like. Am I wrong for being mad?