Doesn’t want to be with me

It was the boy I loves birthday and I threw him a little party at my house just for the two of us. He looked like he had a lot going on when he showed up to my place so of course we sat down and tried to talk about it. He kept leaving things out. He said he had hit a bathroom wall because people were making him mad. So he punched a hole in the wall.

So of course I asked him what the people did to make him angry but he wouldn’t tell me. I’ve never seen or heard him so angry he punched a wall before. He clearly didn’t want to tell me so I just let it be. The rest of the day We had sex and it was great. He hardly says I love you but during sex he does. He fell asleep and I’m not proud to say it but I went through his phone. Found out he punched a hole in the wall because the girl he wants to be with doesn’t want to be with him. Then he told his lady friend that his ex gf(aka me. But we haven’t even dated so idk what he’s telling them) wants to get back together but he doesn’t want to be with me.

My heart felt like it dropped. His phone and computer passwords are my birthday, he actually came to my little birthday party for him, claims he loves me, but doesn’t want to be with me? So when he woke up, I asked him if he even wants to be with me. Like at all. He said he does, and that if he didn’t he wouldn’t be here with me right now or tell me that he loves me. Then again I didn’t want to go into more detail and piss him off even more on his birthday so I just changed the subject. Today’s the next day. I want to text him and tell him I went through his phone but I won’t do that. I should just let him be, right? Or what should I do? My head is so fucked up rn