Bf troubles. Kinda long

So me And my BF have been together for 2 and a half years. 95% of it has been distance. Same state, just different towns for college, job, etc. Anyways when we're together we are perfectly happy. We talk about marriage, kids, where to live, all that good stuff. But during the week when we are apart he gets super mad and sometimes will call me choice words. He even has told me he thinks I've cheated on him. Not true. We've had ups and downs but I'm getting tired if the arguing then him being fine when we see each other. He won't even talk to me bout it on the weekends. Just acts like it never happened. My question, do I stay or leave? I love him and his family with all my heart but being in college and dealing with that stress plus him makes me want to leave. I'm torn. Cause if we make it through it, ik living together would be perfectly fine. Its the distance that's destroying us I believe