Am I wrong?

My friend asked me to come over last night and I said I would but I didn’t feel comfortable hanging out with his roommate. He said that this would person would be there and would hang out with us regardless of how I felt. He basically said I was being rude but I just feel like no one should have to be around someone they don’t feel comfortable with am I wrong?