With the right woman...

I'm pretty ok with having one child, I'm 35 my hubby 40. We were discussing how it could be a challenge if next year we try for a second, afterall our baby needs our attention. Then I said, if he remembers talking about having 3 kids. And his response, with a different woman that would've been possible. He thinks my main job is to look after our baby while he works. I asked him today if he'd like to do the feeding, he said no because he's tired from work. It gets me so mad that men feel the need to pop babies but neglect taking care of them. Should I confront him? I didn't want to cause a scene at the moment so I let it go.