Advice needed? Am I a bad person?

This is going to be long winded so I'll shorten it as much as I can also apologise if this is the wrong place.

Back story: My SO and I have been together two years, he has a child with another woman (3 year old from a 4 year long relationship). We recently discovered that his ex partner is pregnant again in her relationship.

Me and my partner have been talking about children for months, and are trying.

Last night we had an argument because he thinks that if his three year old wants to spend time with both half siblings (when/if) then we should allow it. But I'm not comfortable spending time with his ex partner... And we have his child half and half anyway. He also mentioned that if our child wants to spend time with his current child and he is at his mum's we should take our child to his ex partners house so they can spend time together??

The question I'm asking is am I a bad person for not wanting to spend time with his ex partners child? I was just wondering if anyone else had been in a similar situation.