I DESPERATELY NEED ADVICE

i’m in high school. my boyfriends mom is saying we spend too much time together but we barely had any friends to begin with and when i ask my gfs to hang out they always have to work... i love spending time with my boyfriend we have lots of fun... and his house is more friendly than mine he has a brother and a more comfy and organized home. my house is unorganized because i live with my grandfather and he cannot clean up after himself so i do. he also has a brother to hangout with all the time and when i’m alone it’s me and my tv. should we break up? ... i desperately need advice and i feel so guilty and sad. someone please comment advice