stay or get back at him?

my boyfriend has changed alot for the better and I have made a lot of progress as fair as trusting him...I love him alot and I know he has a good heart and all but I don't know how to feel about this. we have been together for about 3 years and this did happened right after I met him. I never questioned him about why at "this one friend" never wanted to meet me or invite me in he would leave me in the car for hours. never too long but always came to check on me. well one year after my this started happening. me boyfriend lost his ID and needed to pick it up from him ex . so I said yeah sure why not! and he pulled up to the house he use to parking me in front of over a year ago. at this point I know that he was with he's ex every time and we pulled up there. I can't help but think why lie and say she was friend ? what was he hiding ? i know it wouldn't make me feel better but should I try it do the same ? and leave him I the car for hours as I talk to my ex?? or just be done with it all an move on??