Boyfriend doesn’t want to celebrate my birthday

This is the first year that my boyfriend and I will share our birthday’s together. His birthday was earlier in June and he really doesn’t think of his birthday as special. I knew this and he told me that he didn’t want anything “materialistic”, but it just didn’t sit right with me doing nothing because I care about him a lot. So I decided to make a simple meal, bake a cake, and tell him happy birthday which he appreciated. BUT now my birthday is coming up. Ever since I was little I’ve always cried on my birthday because no one made it special which he knows about. So this time I really thought that it would be different and that tradition could finally end, but we were talking the other day and he said “I hope you don’t expect anything for your birthday”. It honestly hurt my feelings a lot. I understand he isn’t big on birthdays, and I’m not trying to push him to do something he’s uncomfortable with but I just wanted to feel like I mattered to him and that he cared. It doesn’t have to be materialistic, I just wanted to spend time with him. We just argued about it and he says the max that he can do is tell me happy birthday and that’s it. Am I disregarding his feelings about birthdays, or should I feel validated in the way I feel?