Husband agreed to try for #2, pulls out! WTH

So my hubby is very hard to get to agree to anything. He kind of needs a push. Whenever I talk about another baby he just says, "ok." I got my IUD out 2 weeks ago which he knew about. We had sex last night and he pulled out...I was so irritated I didn't say anything at the time. I need to address this with him but want to be very gentle. I know he is scared to have another baby..our first was a surprise. He is 2 1/2. It's not an option for me to not have any more babies and he knows this. How can I push my emotions aside and see his side...