My husband and his kids

I have been with my husband for 13yrs. I helped him raise his children along with mine. We have children from previous marriage. However his kids are in their early 20’s. They worked full time and live with their mom. they moved back with the mom after they turn 18. We recently purchase and investment property but now his kids want to move in to the property. I disagree because my husband now wants to charge less than what we had agreed on. I told him multiple times that this is a bad idea because his kid have never been on their own and never paid rent. I don’t want to end up with two mortgage payments. I have seen how his kids depend on him. My husband thinks I’m not being fair. I just feel I already helped enough and I’m getting older and one has helped me my entire life. Him and his kids moved in into my house and it was fine but now I feel like I should get my return on investment back. Am I being selfish?