Help!

My husband of 4 years has told me he wants to have a family and we originally planned to start trying in jan and jan came around and we decided then on april and now he is saying may. I have an iud and I am having it removed tomorrow and he hasnt said anything about it he is even dropping my off to my appt. i me tioned getting condoms u til he is ready to start and he said no and I asked if we should pull out and he said we will see what happens. I feel weird because I feel like I should get his full consent before this. I dont want to feel like ive tricked him or anything. Everytime I talk about having kids he shuts the conversation down but said that he didnt want to use condoms to me tonight. Im just a little worried because I realllly want this but he hasnt actually said yes. How do we start trying if thats his response? Also keep in mind that he finds it hard to make up his mind. I always as his wife choose what we do and how we live and he likes it that way. He was brought up the youngest son and with all women (his mom and sisters). I dont want to have to make this decision for us too. I want him to have equal say because its important. If we have sex without protection do you think I should feel bad about it? Im worried. Thanks.