Confused please help!

My boyfriend always makes jokes saying that I have men come over when he is not home. He even said it before he left to go out of town for two weeks. Then had the nerve to say it again when he got back. I am very faithful to him and it hurts that he would say that. So I asked him what you don't trust me and he said yes I do and I said well could you please quite saying that I am seeing other men because he hurts and I feel like you really don't trust me. We used to see each other right before we became serious but I ruined that and we went our separate ways because I didn't know what I wanted at the time and I was talking to other men. So he stopped talking to me for awhile and blocked me from his life. Which makes sense and I became depressed cause I missed and realized that he was what I wanted so we finally started talking again and have been together for 7 months now and we have a baby on the way. I think he is just very insecure because of what I did in the beginning.