am I controlling? or thinking illogically?

Yesterday my SO mentioned hanging out with a female coworker because she asked. Normally I'm not too worried about it, but it'd just be them alone. She recently found out that her boyfriend (I'm pretty sure NOW ex boyfriend) went to jail for shoplifting. I asked him what they were going to do he told me that she said "idk walk around and stuff" I told him I thought it was weird. I actually mentioned it multiple times. Anyways. Today he told me he might but then never seemed to get a text from her so he said he wasn't going to. Then he told me she texted him "I'm ready to hang out" and he told me he was just going to ditch or make up an excuse to not go. Like an hour later, out of the blue he decided to take a shower and then when he gets out he starts spraying a shit ton of cologne on. I asked him why so much and he told me it was because hardly any would come out in one spray (BTW we were on FaceTime this whole time). I know that's bullshit though because I've sprayed the same cologne his/my sweatshirt and let me tell you. It's strong af. I asked if he was now going out because usually he doesn't do a full routine unless he's going out and he said he was. I told him I had to go take a shower too and he just hangs up without even saying goodbye or anything. I was kind of already upset because I mean.. he told me he cancelled his plans for me and then kind of went back to cancelling cancelling his plans. Now, I'm pretty sure this is the same coworker who's always tagging him in the dumbest shit on facebook and I guess she had no idea he had a girlfriend until he straight up told her. Or "had no idea." The thing that pisses me off the most is literally 3 days ago we had one of the biggest fights of our 18 month relationship. I told him a lot about my past and a lot of psychological struggles I've delt with. His reaction made me feel completely rejected (totally for another story). It ended in me telling him we can fix it, but he's lost a lot of my trust because of how betrayed I felt. Am I in the wrong for not wanting him to hangout with this female coworker alone? I don't know why, but I'm just not comfortable with it and I can't help that. He's told me before he would never cheat and I've believed him, but I just don't feel good about this. It gives me this sinking feeling in my stomach, or maybe I'm just being a batshit crazy jealous girlfriend... I'm sorry this is long too haha.