Trust issues fucking suck

Me and my s.o have had issues we're just getting over. I found out some things about a year ago, no cheating, but keeping conversations with females behind my back, "as friends". He's too fucking friendly and I hate it. My mom told me to work it out for my son so I've been trying to, but one year later and things still aren't the same. I'm not affectionate towards him at all and he points it out all the time. I don't hug or kiss him as much as I used to and right now I feel like I'm just putting on a face to make it work. I feel like I built a wall up and I'm afraid he'll hurt me even worse next time. Which is why I'm keeping myself at a distance. Everytime I see his phone on silent I instantly think wrong. Or a female being added on social media. The distance just keeps getting bigger. We have two kids and feeling like this sucks.