I think I took to to far and now I feel bad ...

This is weird I know but sometimes my boyfriend and I will bug each other . We are 22 and 25 are both extremely faithful and have no trust issues . We've been together for 4 years ... but sometimes we will bug each other and when one of us is texting someone the other will be "whose the side piece" etc . And he just went on a trip with his best friend to the mountains and got back last night . He came over today and we were cuddling and the joking around started . He started it by saying "hmm who you texting you got a new boyfriend while I was gone" and I was like " yeah maybe since you spent $500 on a snowboarding trip with your side piece and can't even buy me flowers" I was completely joking around , I'm not materialistic and don't base his love for me off of if he can or cannot buy me flowers.. he got really sad after and I was completely joking around . Now he keeps saying how he wants to take me on a nice date soon and that we need it and keeps going on and on about it and how he spent so much on the weekend when we barely go on dates . I'm honestly don't mind sitting in and cooking together and doing free stuff so he can get a weekend away with his friend (who he barely sees) and de stress a little ... he's in his last year of physiotherapy  and works so hard and deserved that weekend ! And I don't care if he spent a dime on me it's how he makes me feel not how much money he spends on our dates or on me . I feel so bad that I made him feel like shit .. I apologized and explained to him I didn't mean it and he keeps going off about taking me on a date and that he feels bad .