Insecure...

So a few weeks ago my bf and I were sitting in the back by the fire. My sons sister (who stayed the weekend with me) came out back and said "my dad is about to come pick me up." I said ok. So my bf and I sat outside for another hour until the fire went out. We came in the house and went straight to bed. The next morning when we got up to go eat my sons dads car was still parked in front of my house. So confused by it I went and looked in the guest room and in the other living room. He was sleep with all of the kids. When he got up he apologized for staying and not saying anything because it was late and my oldest let him in. But he said his car wouldn't start. He called AAA and they towed his car so that his computer could be reoprogrammed. My bf leaves and sends me a text two days later telling my that I had him fucked up because of the fact that he stayed at my house. Now mind you I had NO clue that he was there. Hell he didn't even know he was there, until the morning. We came in and walked through my bathroom to get to my room which is on the other side of the house far away from the other living room that my kids and their dad were in.