What should I do?

So my boyfriend & I have been dating for 11 months now and recently I’ve been at his house almost everyday with my dog due to me preparing to move out of my current place and into another place.

My roommates are extremely rude and racist ( whole another story)

So my boyfriend been allowing me to go over there. I always clean after myself, I’m very quiet as I usually just read books in his room.

I bring my own food and sleep there sometimes but 100% I shower, clean my own clothes and all my belongs are at my houses . I have a very small bag I bring with me to his house + my dog who doesn’t make noise either.

I wanna mention we’re both in our 20s and work at the same place.

His roommate who’s on the bottom floor, with his own little apartment ( bathroom, wash+dryers and basically kitchen) in the basement also works with us, so I tend to see him at work plus occasionally the house but rarely as I don’t leave my bf room.

Well his roommate is also in his 20s saw me come by the house today and waited til I was alone and basically said he didn’t want me there, that I need to stop coming on my own because my bf wouldnt tell me to stop coming that he’s tired of me and my dog and he basically doesn’t like me.

I did leave and I’m not coming back but my bf was extremely upset because he pays more to live at the house, that I never make a mess or noise and it was inappropriate for the roommate to say something to me alone .

He also mentioned that the roommate brings random girls everyday to the house and he never says anything.

My bf really wants me to come to the house still but I told him out of respect for the roommate I don’t want to come anymore.

Is that the right thing to do?