Can't talk about infertility

Ch
So I have a weird problem. I've been married for 12 years, and have an 18 year old stepdaughter. My husband and I are currently ttc but it's like it's this huge secret. We get asked all the time why we never had kids or when we're having them and his answer is always the same "we can't afford to properly take care of a baby". It breaks my heart to hear him say these things to other people because we are well off and can afford it. Not to mention it's like he's too ashamed of me to admit he wants to have a baby with me. 
His daughter doesn't want us to have any kids and asked us when she was little not to. She has siblings at her moms but loves being an only child at our house. I can understand her feelings but at the same time, she's away at college and old enough now to understand what she's asking of me. 
So I guess if I do get pregnant we'll have to pretend it was a big surprise and wasn't planned. I want my baby to know we wanted and planned them. So frustrating to not have anyone I can talk to. I do have a couple of close friends that know but if he knew I talked about it, he would be upset.