What should I do (long)

So my bf and I have been together almost 2 years. Things have been really good with us lately. I've been getting closer to his son, and that's a big deal because the only other person who's met him was his ex wife.  After his ex wife and he divorced 3 years ago, he started dating this girl X. They've known each other for years. Well he and X broke up and then he met me and we've been together ever since. About 1.5 (Oct 2013) years ago, when we were only together for about 6 months, I saw he was texting this girl and leaving the room when he was on the phone with her. He lied and said it was his male friend. I didn't believe it and he said ok it's this girl X and she's just a good friend. (Never told me he dated her before he met me). So I let it go, then one day he was in the shower his phone went off and it was her, I read their messages and saw that they were saying "I miss you, I want you so bad, I just saw this on tv and it reminded me of you" etc etc. I freaked out and left and he said he wouldn't talk to her anymore and that she was moving to Boston so I have nothing to worry about. So that was in October 2013 and a few months later in December he and I broke up. We got back together in January and I guess when we had broken up, he started talking to X again and "tried mending their relationship". So we were back together but he was acting weird. (I lived 2.5 hours away and I would drive to see him on the weekends). Well I would drive down on a Friday night after work, and he would make me leave Saturday afternoon, every weekend. He said he had work. I later found out he would have me stay Friday night, and X stay Saturday night.  So one time I decided to drive down and not tell him. He was still working. I guess X had texted him saying she was going to come over and wait for him to get home. So his roommate told him I was at his house and he was fine with it until X texted him saying she was coming over, he FLIPPED out and forced me to leave. I had nowhere to go and couldn't drive 2.5 hours home at 11:30 at night. But he chose her over me and he made me sleep in my car the whole night. (This was a year ago so it was February). It all made sense. I found Valentine's Day cards from her hidden in his drawers. We were supposed to do something for Valentine's Day and last minute he sad he had to work. I find out he and X Went on a little vacation together. I was just so devastated. He completely turned on me.  I guess he didn't want her finding out about me, which is why he told his roommate to literally kick me out. But she did and she broke up with him. He begged her back while he told me to F off. Eventually he took me back since she wasn't budging, but he still secretly talked to her and anytime she would budge and say ok let's hang out for the night, he would immediately cancel his plans with me. So this all went on for about 3 months (the secretly dating us both). Then finally he and I mended things and he and X didn't talk anymore. This woman has 3 kids, and can't have anymore. My bf wants his own family, which is why he ultimately chose me after this drama all died down. She doesn't even seem like his type, she's thin, not pretty, and acts like a dude. She has a good personality though, but he even said she's more like a friend. His family loves me, I'm so close to them. they don't like her. All his friends like me better, and tell me I'm better for him and I have so much going for me and that I'm good for him. (I'm younger-21. He's 35. She's like 32). So it's been a year since all this happened. I've forgiven him and we've been good. I live with him now, and have been for 6 months. I went on his phone and blocked her number and deleted it, so he can't talk to her anymore. But he somehow unblocked it. So now this is the point of my post (finally lol)-- Yesterday I was on his computer doing work and he has an apple computer with his Apple ID hooked up to it I guess, so his computer receives his text messages I just found out. So I'm sitting there and a little message pops up and it was from X. telling him to call her she's a mess her life is a mess and he said "I just tried calling you are you ok? I'm worried about you".  I called him immediately and told him what I saw and he said she texted him first and that he doesn't care he won't talk to her anymore, And that I can look at whatever he wants cause he has nothing to hide. Then I see this morning he changed his computer password. So I called him and got upset and he gave me the password. Its just so stressful. I've told him repeatedly that I DO NOT want him talking to her. Just friends, or not, I don't care!!  It started out with them talking as "just friends" a year and a half ago and it progressed  into 4 months of him cheating on me. And he even said back then "we were just friends but it just progressed into more, I never expected that to happen". So this is why I don't want them having any contact. Period. I don't care if they're "just friends". If he's going to be with me I don't want him talking to her or seeing her at all!! Unfortunately she lives in the same town so they bump into each other sometimes he says. She only lives a few minutes away. Which makes me nervous. Am I asking too much, asking him not to talk to her anymore? Please, tell me if I am being crazy. I know I have insecurity issues now because of all this. And I wish I could be the type of girl who says "sure talk to her, I don't care". But I don't have that type of confidence. It brings up some really bad memories and feelings seeing him talking to her again. I don't want them being friends. I'm afraid it will progress into more again. And I don't see why he needs to still talk to her. Given the circumstances of everything that happened last year, I feel like I'm justified in asking him to never speak to her. But he makes me feel like I'm crazy. Really, am I asking too much? Or should he not talk to her anymore? Sorry this is so long and confusing! & please don't say to leave him, I'm trying to work this out but I'm just looking for justification from other women's standpoints to see how they'd feel in this situation.