Am I crazy for not wanting sugar?

My son's first birthday is coming up and I was discussing the plans with my SO's sisters and the topic of cake came up and I told them he (my son) isn't going to have any. Sure, we might have one at the party but I don't intend him to eat any of it and they said I was crazy and mean and "of course he deserves to have cake on his birthday". I just think it's so unnecessary to introduce sugar before he's interested Our pediatrician recommends waiting with sugar for as long as possible. I know we can just let him eat it that day, but I just don't see the point, other than it might be cute for us adults to watch. We're also struggling with getting him to eat "normal" ( aka not breast milk) food so I'm afraid giving him sugar will make this even worse. Am I overreacting? What do you think? Cake or no cake?