Who to believe

I have two kids, 2 and 4. After I split from their dad, I brought a new guy into our lives. I was so happy to see how great he was with them and they both became attached to him very quickly. I loved our new little family. Months have gone by, we all live together for about 5 months now. All of a sudden the past 2 weeks, I've noticed changes in my 2 year olds behavior when around my boyfriend. He's not talking yet, so he can't tell me anything. But he is obviously scared to be around bf. I have not seen anything unusual. I have never known bf to be abusive. Today 2yo came running out of our bedroom crying. I had just walked out seconds before so what could have possibly happened within 15 seconds of me walking away? But my 4 year old then asked why did he hit him? I took both kids downstairs away from bf to ask them if that did happen. They both nodded yes. Of course I would do anything to keep my babies safe. It just doesn't make sense. Kids like to make things up that don't really happen, but the cry isn't fake. I also wonder if anything has happened at their dad's house when they are there. He does have abusive tendencies, though not towards his kids that I've ever known. But I also don't know any new people that their dad could bring around or what my kids see over there. Both kids have also been unusually quiet the last 2 times i picked them up from their dad. This is such a hard situation. I have already brought it up with bf and he denies doing anything bad. I so badly want to believe bf. But my babies safety is top priority no matter what. I hate to wrongly accuse someone. But how do I know for sure what is happening if I don't see it myself? Broken Mama here.