Stressed now.

So ive been seeing this guy. Casually although we have had sex. Things have been going pretty well. We happened to work together but different campuses different shifts different departments. Im very close with one of the guys on my shift and campus but his department. Tonight we were talking and he was like "oh no not him. People dont take him seriously they think hes a big joke and no one respects him" now i tend to take other peoples opinions very seriously and now im worried if he says something that people will think of me that way, ive worked very hard to get where I am and people think im smart funny and nice. Should i continue seeing him i do like him, but i dont want to be thought differently of. I know its horrible of me.