Help!

So yesterday after 2 weeks of not doing nothing more than kissing the guy I've been dating for the past 6 months, we finally have the opportunity of going to his place. I have al ready been there around 10 times, and its always the same, we start watching a movie while we eat dinner and after that we make out. I'm a virgin he's not, and he has asked me twice to it but I said no in both times, he respects that. Well yesterday after we finish eating dinner we start kissing very intense and he was taking out my shirt and in that moment someone tried to open the door. Then they knock it several times, so we stop kissing and I go to the toilet, when he opened the door it was his younger brother who is older than me anyway. After that I didn't want to do nothing more, because I felt that someone could open the door and watch us. I'm 18 and he is 24. I get a little panic about someone finding us out, he respect that and we just stay in bed naked, anyway I'm so frustrated because I have been very horny the past weeks and nothing happened more than kissing. We tried to make out after that but I wasn't in the mood anymore :((

Is it normal that yesterday I didn't wanna do nothing more? After an hour of that I give him hj but just for him to get pleased, it wasn't his fault. But I was like "ugh".