Ppl don't like us together?

Has anyone ever have to deal with being in a relationship with someone but people not liking the fact that you two are together? I'm currently dealing with this bc ppl are so use to him being with his ex that he was with for two years ... It hurts bc it's like I'm in his life now, she cheated on numerous times so like what's the problem? Why can't they be happy with us? I'm guessing bc I'm not her, should I care? It isn't his family, they love me but it's others that he associates with ... So my question is, should I care?? Or just continue to love him and go about our relationship...