Would you feel uncomfortable in this situation?

For background purposes: My husband is very close to his father. They live in the same neighborhood as us. They are both builders/mechanics and do a lot of work and projects together. He goes over to his house very often to work on things together. More than I believe is normal, even given the shared work and interests they have. His parents are pretty “free spirits”. They do whatever they want basically. My husband lived with them up until we got married 1.5 years ago. So 24 years of his life. He would have moved me onto their property, but that was a clear no from the beginning. If that were the case, as much as I love him, I never would have married him if it meant I had to live with them. I don’t need to live under his parents rule after already living independently on my own as an adult for 7 years. So anyways, we have our own home now, in the neighborhood he grew up in. A family friend is selling it to us for an extremely amazing price.

Here’s the thing, he had a live in gf at his parents property for 2 years. His parents open and willingly moved her mom and brother onto the property while she lived there. Brought in the whole family. After ex gf cheated on him, he kicked her off the property, but she would always be there to hang and visit her family there even after the incident. Even the ex gf dad drops by and visits throughout the year. So they kept the mom and brother and call them family. MIL talks about ex gf sometimes and I’d rather not hear it. I truly wish they had a clean cut and I could have my own In laws and my own family. I’m a very private person and I feel like I can’t do anything without everything being involved with ex gf family or them having to know about it at some point. Though I don’t think that she really comes around much anymore and I’ve never met her before. Her brother moved out but is coming back soon to move back in. They call him “their son”. Her mom moved out for a little bit because she caused issues for my husbands parents in their marriage and then they brought her back in to live there. I believe it was a bad choice to bring her back. Would you feel uncomfortable having to be around these people? I feel like I’m stuck in the twilight zone or an episode of Dr. Phil. This isn’t normal to me. Not everyone is meant to stay in your life forever.