When to expect the unexpected ?

I've been with my fiancé for almost 2 years, we have a baby boy on the way in June this year. I have a 5 yr old from previous relationship and he has an 8 yr from previous relationship. But I can't get the fact that with my personality and self respect I don't hang out with none of my ex's nor baby father period but one of his ex's is his brother fiancé cousin and they all are okay with hanging out together , in his eyes there just friends what they had didn't work but there friends, I don't feel comfortable hanging out with anyone ex's who I'm with I won't hang out with my ex's and bring my boyfriend in that space its not me..I mean she has a boyfriend but I don't trust females. I m very alert to anyone , When we go visit his brother house and his ex shows up or there I feel awkward and get to thinking about what did they have when they where together??? One night we planned to go with his brother and fiance to her friend home going first time I actually met her is when she came to his brother house, I felt as if it was a set up like I didn't know she was coming over, then I agreed to go along to the home going we had to stop home to get dress not knowing who all going , find out she's coming alone let along parked outside my house like if I wanted your ex to know where I live I would've said something, I was mad at that, we get there everybody having a good time but I'm not cause I'm thinking otherwise, he knows me , he knows when ain't feeling shit my face expression change and you'll know, he could've told me that night look if your not comfortable we don't have to hang out with them , I explained to his sister n law they I felt and it was awkward and found out bf was there the whole time, my fiance didn't explained it to me in any kind of. It would've been cool if I knew. But I'm the type I don't want to get to know your ex's nor hang with them like we one big happy happy that's not me cause I don't want you in that position with my ex's, Am I insecure ? Should I worry? I approve of friends but what's making me sometimes get in my crazy attitude mood is because one time I went through his phone and him and one of his other friend was texting inappropriate things even if he didn't mean anything by it he says , I don't talk like that nor do I approve of talking like that cause it would be different if it was me, some of his female friends he have dated, they had sex etc so I'm wondering because I know how a female can feel dealing with a relationship and trying to become friends. I trust him but I'm wondering today why does he has a lock on his phone, I try not to think of little things as big problems