The term "Baby Daddy" Applying to Black People

Since the day I started announcing my pregnancy at work and at school, I've noticed that people refer to my child's Father as my baby daddy and not my boyfriend or my fiancé. They'd ask questions like "So is your baby daddy excited?" As if they are trying to ask we we together without directly asking. I find this quite offensive. I correct them by saying " WE don't use that term baby daddy, and yes we are both very excited." Does anyone else experience this?