Boyfriend Troubles and Sex

Recently my boyfriend was telling me a story where a guy told his girlfriend that he was thinking of another girl when they were having sex.

He asked me if I ever have and I honestly said yes. I have thought about my ex, the only other person I have had sex with twice. I was not fantasizing about him or anything. He just came to mind at the wrong time. My boyfriend is really upset and telling me that it’s disgusting.

I asked my mom and she said it’s human nature for us to do that and that it’s okay.

It’s not okay when it becomes a frequent thing but once in a blue moon it happened and now I don’t know what is going to happen between my boyfriend and I.

Any advice?