Daddy Issues...

So I need some serious help... my now x husband has told me he wants nothing to do with this baby and wants to sign away his rights, but expects to maintain a normal relationship with our 8 yo son... my first instinct was to tell him no, he doesn't want to go with him and I'm not going to Force him to continue a relationship with him bc I refuse to let one child feel unloved and left out bc he wants brother and not them (which I know I have a while till I have to seriously worry about), but he told me that wasn't right, I don't have to worry about that any time soon, and as a parent my job is to put my foot down and tell my son that even though he doesn't want to go with dad, he needs to bc he only gets one and it's his dad... which I can understand that... but I guess I just need some reassuring if I made the right call (if you don't want both, then stay away from both), or should I let him keep the relationship with our son and abandon this baby?