Who's right ??

So just q random question. Okay me and my husband went to a little kids party and it was time to eat the cake. The lady came over handing out cake and when she approached me to give me a slice I kindly told her "no thank you " because I didn't want cake. Whoever my husband told her yes even tho he didn't want cake. He says that I am wrong because I should take the cake even tho I'm not gonna eat it because of manners but I feel like there is nothing wrong with saying no politely because I don't want to waste the cake. So we are trying to figure out who is right ? What would you guys do take the cake or no??