Dealing with S/O and his family?

As of the last few weeks they have been telling me that i need to move closer to them away from my doctors and my family. Saying that my family cant give me what i need. I dont feel comfortable around them for long periods of time. Plus we cant afford a place of our own right now. I'm 18 w 3 d and dont want to change my doctors. Am i being irrational? My S/O told me last night that i dont think of us as a family because i told him to take a weekend to himself and go have fun. Am i not doing something right? I am asking for ways to deal with this without stressing out or telling them all to go to hell.