Having trouble with middle name

I am pregnant with my third boy. We already agreed on the first name. My first son has my husbands middle name (my husband shares the same middle name with his dad and brother), this baby I want to include my family since I don’t have a daughter I can’t use my name or my moms name. I wish I could do my dads name but I am not a fan of his name, so I want to do his middle name after my little brother who I am very close with and means the world to me: My husband got so mad saying that, that’s not my brothers kid and that it should be his first name (my husbands first name) as our baby’s middle name. I find that so unfair since he already has our oldest son with the same middle name as him and they have his last name not mine. Why can’t I have something that means a lot to me? I want at least one of my kids to have something to do with my family. Not fair for him to get two kids that would have his name and none with me when I am their mom and do every thing for them. Am I wrong or is he wrong?