self conscious

So my bf lives in a hosue with 5 other guys. He just moved in there since college started. Well we have been dating two years and havent had sex, which is our personal choice. I know many people don't understand this but we are happy and thats what matters. I am very self conscious/scared to go over there and have someone make mean comments about this. My bf and i are ling distance so when we see each other its usually a month and we just want to be alone in a bes basically. Its not like my bf broadcasted this to them but they are all close so know. Does anyone have any advice for me to get over this?