Boyfriend staying alone an hour away... Should I be mad?

My boyfriend and I have been together for 5 years and go to the same college an hour away from our hometown. This week is spring break, and we're about to go out of the country on a trip together. We're leaving Wednesday. He told me last week that we could stay at his house since no one will be there Sunday night (tonight) and Monday night, then stay with our parents Tuesday night. This last Friday night (3 days ago), he told me that he wanted to stay at his place, an hour away in our hometown, alone because he wanted to hang out with his friends. He said he wanted me to be there, but he knows that I don't really like hanging out with these friends because they smoke and I just think it's kind of boring (which is true). Long story short, a three day argument ensued, and today when we were deciding on him staying here, me going with him, or him going alone. I told him that if I go or if he stays here I'm going to feel bad, like I'm keeping him from seeing his friends he never sees, or like I have to be there for him to hang out with his friends... I wanted him to go because I knew that's what he wanted to do, and it was really too late for me to pack for the next week and go anyway... he told me he either wanted me to go or he wanted to stay here. He bawled crying and got really upset and said he felt awful for hurting me, etc... He ended up leaving because I basically forced him to go, but I honestly do feel kind of annoyed and upset still. Do you think I'm wrong for feeling that way? I totally trust my boyfriend, but do you think I shouldn't? Has this whole thing been blown out of proportion? 
***Basically my bf asked me to go stay with him for a couple days at his empty house an hour away, then uninvited me saying he wanted to see his friends. I got mad, we fought, he felt like he was letting me down/upsetting me, so he didn't want to go, but I didn't want to be a psycho gf and say that he either had to have me there or not go, so I told him to go and he finally did. Even though I promised him I wouldn't be upset, I kind of am... Is that wrong?