am I wrong?

So my hubby asked if he could go to California for a 5 day vacation with a few of his friends (they are all single).  At first I said no and that it's not fair since I am stuck at home with 2 kids (2 year old and a 2 1/2 month old who was a premie) and I spent a month and one week in a a different city with my 2 year old taking care of my son in the NICU. And we never ever go anywhere or do anything fun plus I've never been on vacation. I asked him why we all just couldn't go on a small vacation and he said it's to expensive. Then he went on and on about how he works and does this and that and how this seemed like a good opportunity to have some fun. 
Then I felt bad so I told him he can go if he really wants to so he booked his ticket and is going beginning of next month. 
Am I wrong to still feel sad, upset and a bit jealous that he gets to go even tho I said he can if he wants to.