should I give him a break??

So this guy is my best friend I have known him since my freshman year of high school and I'm now coming to an end in my junior year. I really care about him and ally have ever wanted was the best for him. He moved the summer we got out of 9th grade but he only moved from Alabama to Georgia so it wasn't a long move. Me and him have had feelings for each other and he took my virginity. We are slowly starting to drift apart and it's literally breaking my heart I want to cry. He's the only real friend I have. The last friend I had we drifted apart (she was a girl) and it ended very nastily. I don't want our friendship to end because that's where it started and I do truly love him. I don't want to take a break from him and him leave because when I have suggested it when we had fights he said he wouldn't come back for me and that if I wanted to leave he would just let me which makes me upset. I don't know what to do I want my best friend back but I don't want to keep him from growing and things and he says he's worried about his job and his grades so I don't want to get in his way. What's the best thing for me to do ?