Boyfriend troubles

Okay so, my boyfriend and I have been dating for over two years, and at the end of the two years, he broke up with me twice. Each breakup was over text. So we got back together like a month later, because he said he missed me (this was also over text). He said he made a mistake and it's the first time he has cried in a long time. I mean I'm happy I'm back with him, I really am in love with him, but I conflicted with my feelings. I feel like this relationship is just physical attraction, and we never talk about our feelings. He sometimes just ignores my texts. We recently just had an argument these two girls (his best friends) going to his house at midnight, while his mom and brother were on vacation. So he was alone in his room with two girls in the middle of the night. I said I trusted him, but I told him I wasn't comfortable with that. I mean why couldn't they come over in the day, or when his family is home. Then he started accusing me of thinking he was going to do something with them, and I didn't even say that! All I said was that I was uncomfortable with two girls going in his room in the middle of the night. I really love him but I can't help how I feel. I feel like we're drifting apart after the breakups, but I don't want to lose him again and I don't know what to do anymore. He is my first boyfriend and we are both seventeen. I was a total mess the first and second time it happened, and I just don't want to be like that again. Please help, advice would be greatly appreciated.