Insecurities....what to do

I was abused been through hell, cheated to lied to used and raped. Bullied at school and my older sister. Whose my half sister. I warned my husband about my insecurities and how I am. He has been completely okay with it and I have been working so hard to give him freedom. I had this thing about his phone that he couldn't take it to the bathroom because I would feel like he looked at women and do stuff. Well, I worked so hard about getting over that. He asked me today about taking my phone with him to the bathroom and I said he could take his. He took it and came out and got MAD at me for letting him take his phone! He said I never use to do that. Well, now I don't know what to do. I worked hard to help not only him but me and he gets mad. Ugh!