Feeling alone and defeated at TTC

I've decided today, one day before AF is due, that my husband doesn't want to have a baby with me. When we were dating and first got engaged he told me he didn't think he ever wanted another baby until me and that he wanted to start TTC immediately upon getting married. Now, we've been married for almost a year (April) and it keeps changing. Sometimes he talks about it and "what if we have a girl" and other times he doesn't communicate with me and when it comes up our plans have changed with him and I don't feel he truly wants it as much as I want to have a family with him. I understand he is most likely stressing because he wants us to get a house first and he wants me to stop working once we have our first baby together so all of the pressure is sitting on him financially I'm sure and it's probably scary but he likes to spend money on things that are not needed and I'm very frugal so I'm ready to save and get a house and he wants to keep spending freely but it's pushing back our plans to get a home and start a family. Honestly, it just hurts and confuses me. I'm 35 and have no children of my own and he has an 11 year old son from a previous relationship. I'm not sure how to say all of this to him without him getting defensive. When I bring up how I feel about it, he gets down and says that I'm hurting his feelings and making him feel like he is letting me down. 😔