Am I controlling or should a relationship have limits?

I'm in a long distance relationship with my boyfriend. He works a lot & if he's not working he's either at home talking to me on the phone or hanging out with his boys. Just recently I went to vegas for my 21st birthday weekend with my FAMILY. (Boyfriend couldn't go because his manager wouldn't let him miss work) So while I was in my hotel room getting dressed, I was on the phone with him while he was at the mall shopping. Then a few minutes later he tells me he's going to smoke weed with a female friend of his that he's known since high school. But I don't know, for some reason it bothered me and I had him drop her off before they had a chance to smoke. 
​I trust my boyfriend 100% and I know he will always be loyal to me. (I was friends with him for 5 years before we started dating) 
​But, the reason is because I'm not comfortable with him hanging out with another female unless I've met her in person. 
​Yes, I let him hang with his boys and, of course, family whenever he wants. But when it comes to other females, I'm not okay with it. Don't get me wrong, I'm not insecure. I just feel like there should be boundaries or he should at least make sure I'm comfortable with it.
​And of course, he got upset and said I'm crazy and controlling. And didn't even consider my feelings at all, he just got upset instantly. 
​Do you gals think I'm controlling or should there be boundaries? 
(He's only been in one serious relationship before me & that was in high school, so he's fairly new to what should and shouldn't. We're both 21)