Indifference into heartbreak

So I have two kids and my husband has one. When we got together I told him I did not want anymore. He said ok, knowing he wanted more. So we're married and he starts working on me to have one or two more. I'm in my 30's and I'm thinking, no dude. But eventually I give in and say ok to one more. But honestly I'm not excited about it. Obviously I'll love my baby, but I'm not looking forward to the process. I love working and he wants a stay at home wife/mom. I just don't want to start over. My youngest is about to be 6 my oldest is 9 I thought I was done. So anyway month one TTC I'm thinking this will be stress free for me because I'm not looking forward to it anyway and I don't care if it happens right away, whatever. I take a test and that negative line really hit me. When did I become so excited about this? Now I feel that heartbreak of a negative test and it's caught me off guard. I was originally doing this for my husband who's always wanted children since he was a kid and who doesn't get to see his daughter but every other weekend. Who didn't get to be apart of the pregnancy or delivery due to his AWFUL ex-wife (she didn't even tell him she was in labor then refused to let him see her till she was 3 months old at Walmart). He's never been able to feel like a father or have an active role in her life till very recently, she's 4 now. I said yes just for him but now I feel like it's for us both. I have no idea where it came from. If I was denying wanting more or if I didn't realize till I seen that negative test. Life is such a rollercoaster.