Boyfriends birthday

So me and my boyfriend of almost a year and a half have been saving a little bit of money together. I told him we can’t touch it without asking each other first especially if it’s a bigger amount. Well I talked to him and asked if it was okay if I took out $20 for something I wanted and he said yes. Well a week ago he took $80 out and spent $20-30 of my money on himself for action figures. Didn’t tell me or anything. He said he assumed it was okay since I took out $20 of my money. And I said he should pay me back or put the money he spent back into the saving thing. He never did. Today’s he birthday and since I don’t have a lot of money I usually spend about $60. Well now I’m only going to be spending about $30 on him since he took some of my money without telling me and spent it on something he didn’t even need. Am I wrong for this?