Is it normal to think about other boys?

I love my boyfriend so much, but whenever I'm alone my mind starts to wonder and I think about other guys I know sexually, one is even his brother. I keep these sexual fantasies to myself but I feel so guilty about it. I don't love him any less afterwards, but I wonder if my sexual curiosity is detrimental? He's the only boy I've had sex with and I think that might be why I kind of want something different even if it is just all in my head. Any thoughts?