Tired of negative input from friends/family

I am 38yrs old and have 13yr old twin boys from a previous relationship. Their father died when they were a year old. My husband is 40yrs old and has a 13 yr old son and an 11yr old son from a previous relationship. We have been married for 6mths. We have decided to start trying to have a baby. I know we are a little older and yes our kids are older. I know it will be like We are starting over. When we have told family and friends that we are trying to have a baby their response is "have y'all lost your mind" "do you realize what you are doing?" It really upsets me. I have waited 12yrs to find someone to fall in love with me and my kids and someone who wants to love me enough to have a child with. We both are responsible adults, we both work, however we don't make a lot of money but we make it. What is so bad about us having another kid? By the way I have 4 nieces that are having babies very soon. Nobody has said anything about them having a baby. One is even 5mths younger than me and already has 3 kids so what is the difference? Ugghhhh