Am I the bad guy here ?!

So my sons 1st birthday is coming up and it’s on a week day so my husband and I have organised a party for him on the weekend which is only 3 days later so all the family can be there to celebrate.

On his birthday though my husbands parents and grandmother are expecting us to take him to see them. Am I in the wrong for not wanting to and just wanting to have his birthday with my husband and I as a family ?

I don’t think it’s really fair that they expect to be involved in every aspect of his life just because they live closer they never let us enjoy his firsts they always want to be involved it was like Christmas time they wanted to be here when he got up in the morning to open his presents with him and I tried explaining to them so many times that the morning was going to be us as a family then be at there house for lunch but that never sunk in until my husband ended up just having to be straight out blunt with them but even after that they want control and input over every single thing it drives me mad

Am I wrong for just wanting his special moments between my husband and I ?

UPDATE

Just in case people didn’t realise we are having a family birthday party 3 days after his birthday so all the family can come together and celebrate ...