Am I being ridiculous?

Kaylynn
So this is a long one please read!...
So my boyfriend has two kids from a previous marriage and he only gets to see them on holidays and for like a month during the summers because they live in NC with their mother and he lives in FL. They will be visiting in two months and he wants to do a week vacation, like a cruise (which can be expensive during the summer) or a Disney get-away. 
I have gone part time at work because my bf wanted me to return to school full time to get my bachelors in education, so money has been very tight for me and I have also been trying to save whatever I can so I can have an emergency fund. 
My bf is in the process of buying a house for us, and between the costs of that, paying 1500 in child support a month, and also paying for his kid's flights here and their vacation, he wants me to pay for myself for anything we end up doing while they are here. 
I took time off, tried to suggest less costly things for us to do, and explained to him my financials, but he said that he would love for me to come with them, but if I can't afford it then I shouldn't go and that he isn't going to "cheapen" his kid's time here by not going on a real vacation. This really hurt me and I got very upset with him because it will also be my vacation that I took off from work and I don't want to put myself in a financial bind. 
Am I wrong to be upset or am I overreacting and acting selfish?