My boyfriends brother...

Shelby
Is it bad that when me and my boyfriend first started dating. I kinda found his brother attractive.. And I had a sex dream about him. And I told my boyfriend about it now he keeps holding it over my head.. Is it bad to fantasize over somebody besides who your dating?