Should it matter if his mom and sister doesn't like me?

I knew this boy when I was in high school , he like me but was younger so I never talk to him. I seen him around a few times and bam one day he became my next door neighbor!! We talk me and his family got along but he does have a child by another women. Me and her had our argument and I left it along , me and his mom got into it and she call her daughter. Yesterday his baby mama started a hole seen his mama made things worse and his sister came. But we didnt fight cause i choose to be the bigger person. Me and him love each other so much and our connection always stay the same no matter if we are friends or in a relationship. I haven't said anything wrong to these people for about 3 or 4 months now. I'm not one to disrespect anybody but I will not stand for disrespect. He has changed and really shows he love me. I just wanna be happy with him. Should it really matter if they don't like me? He said he doesn't care about that he still loves me. I just wanna get some opinions on it. Thanks love ❤