Too soon to try for #2?

I have a 3 and a half month old daughter and my husband and I have been getting baby fever. He keeps asking if he can put his baby in me again but I don't say yes because idk if it's ideal. I want to say yes but I tell him I'll leave it up to him and then he pulls out which kinda leaves me to believe he's not sure of the timing either. We had 2 miscarriages before our rainbow baby and we want a big family. I get really turned on by the thought of him filling me up and get disappointed when he pulls out yet I don't tell him not to. Thoughts?