Can't let go...(sorry for long post,but need to vent)

I think my SO and I are at the end of the road... We've always been off and on for 3 years, but about 7 months ago I moved in with him because we were good, the it started to be he just didn't treat me like he loved me or whatever and we got in a huge fight and I left, moved out. He was so hurt and wanted to be with me and did whatever we got back together a month later(that was the longest we've ever been apart)so it was perfect I ended up moving back in. Been together now for 5 months and have been good until last night. He hasn't been having sex with me for a while for no reason just rejects me a lot, we got in a huge fight last night because he was drinking all day and I saw he was looking up girls he use to talk to and did whatever when he was with me. I was just upset because I don't understand why he even cares or what's so good about them. And that he's sexually been neglecting me and just going back to the same he's always been just not treating me the best(he's not cheating though)but I ended up exploding last night because of all of it. He was mad because I still go through his phone, and he's upset because I should trust him and he doesn't do that to me. Which I agree but what he's done in the past I have a hard time letting go of things especially since he never wants to have sex I just want to see if somethings up because I feel crappy. I don't think he understands that. And now he still won't sit and talk to me. I get he needs his space, but I'm not okay with the silent treatment because it didn't solve anything and makes it worse. I don't want to deal with being treated like this again because I haven't done anything. But at the same time when we're good were great, and I really do love him. I don't want to be with anyone else but him, and I don't want to up and walk away again when I know this time I want him forever, because I don't think it's right and that's what he said to me last time we broke up. But I just feel like he still doesn't want to be with me, even though he said he didn't want to break up but if I kept pushing it we probably would. I'm just super hurt because I really love him. And I'm suppose to be leaving in 2 days for 4 days and I don't want to be like this when I'm gone and be super depressed around my family. I told him I'd leave him alone again tonight, but he needs to talk to me tomorrow and work this out, or I just need to leave as much as I don't want to. I'm just very hurt, and don't want to let go of something I love and care for so much. But I don't know what to do anymore. I feel like it's always the same song and dance, but it also seems to get better each time... Just don't know sorry I just don't have anyone else to talk to😞 And pls no negitive comments, last thing I need right now...