sorry it's long but need help

So I've been having problems with my bf about trusting him... He did stuff in the past and we got through it now I live with him we've been good for 5 months. But I still look at his phone once in a while and he barely has sex with me anymore, and he said it's because I go through is phone and don't trust him. Well I went out of town over the weekend just got back today, he ignored me all day on his night off even after I sent him pictures. My friend said he wasn't home that night(she lives across the street) but I asked him he said he was in bed at 9:30 then left in the morning for a court date which I knew about. Well me being parinoid. I just went through his truck and found a condom.. Unopened. I came inside and asked him, he said he doesn't know it probably came from his other other truck when he took stuff out and he doesn't ever go through his center Consol... Well I've been off and on with him for 3 years, in the beginning he was hooking up with other girls, then supposedly stopped after our first year. Which is when we used condoms for a couple months, then we stopped using them so it's been over two years since we have. And the previous truck he got after a 6mnts-1year we were together then he sold it in October 15, got his new truck.(we broke up for a month in Nov-dec but he said he wasn't with anyone else).Hes trying to tell me he doesn't know where it came from, it was probably from the truck before in the consol he didn't look through it, mind you I've looking through his new tuck consol and NEVER found a condom in there. Until now. I was actually being pretty clam about this, and now he won't talk to me cause he's trying to nap bc he has to go into work tonight for a few hours. Which I get, but think this is more important right now to. I just don't know what to do. I've tried so much and really love him and didn't think he'd do this, but his story doesn't add up at all. And I've just had anxitey keep thinking something bad was going to happen. Now he's thinking of breaking up because I don't trust him, but he's given me reasons, and I've been trying to trust just takes a long time to build up when you lost it a while ago. I've put so much into us and doing everything for the house and I just feel like I failed. And it's so embarrassing going and telling my family this because they know we've always been off and on. And I don't want to lose him I really do love him. But it's so hard when he doesn't treat me like I'm important and when I find that like what am I suppose to think? Am I wrong for doing this and thinking all of this? Sorry I'm just in distraught...