Partner doesn't want a red head baby... should I be annoyed?

My partner and I were watching South Park last night and it was that 'ginger kids' episode. I was laughing all through it at how ridiculous it was that cartman thought red heads were disgusting and evil and my partner was too. In the middle of the episode I asked him jokingly 'hey you'd still love the baby if it was a red head, wouldn't you?' (I'm almost 12 weeks pregnant) and he kind of gives me this look and I'm in complete surprise because I asked the question completely jokingly. He told me he would REALLY prefer that it wasn't. I got a little annoyed and said 'then why are you having a baby with me? I have tons of red heads in my family, both of my brothers are! And you have bits of red in your beard! You know it could be a red head...' and he said he didn't have a problem with red heads he would just prefer it if our baby wasn't one. He said at the end of the day he'd still love the baby. I seriously couldn't believe how he felt.. should I be annoyed?