FWB crisis

There’s this guy I wanted since we met and we recently started talking to each other again. He keeps inviting me to his dorm the only thing is the school he goes to ALOT of people there don’t like me 😩😩 and I don’t want them to see me. So like what should I do cause I REALLY REALLY want him