Is this insensitive..

My brother has cerebral palsy. He’s great and I love him to death. I called my SO to say goodnight and mentioned my brother and how he’s doing. My SO asked me “how did your parents explain your brother to you?”

I said, what do you mean. They never had to explain anything. That’s my brother, that’s who he is. There’s nothing else to explain.

He pushed the subject. I got upset. Why should anyone have to explain your sibling to you?! Why should I have to justify who my brother is to anyone.

SO refused to accept it when I told him that was an insensitive thing to say and I was hurt by it. I don’t know what to do now. I’m so so angry and insulted right now. Was what SO asked me insensitive? Am I just overly defensive?

UPDATE:

I made a big deal out of it. And now I think he’ll break up with me after 2 years together. I feel like the biggest fuck up ever. I defend my brother and now my SO might leave me. Fuck my life.