What to do? Advice

So my birthday is next week and my family wants to take my husband and I out for dinner. Today I told my husband that and he got really upset saying that he already had a surprise dinner planned for me and that now it is ruined, he said we can go and eat with my family if that's what makes me happy. I feel so bad! I don't want to say no to my family or ruin his plans! I already suggested doing brunch with him but he's probably going to be working! What do I do? I just want to enjoy my birthday with the people I love and get them to also have a good time