Married but family doesn't want us to have another

My husband and I have been married 2 years and have a 3 year old daughter together. We decided we want to give our daughter a sibling now so they'll only be 3 years apart. The problem is I'm terrified to try because my family acts like getting pregnant is disappointing. They talk to me like I should only have 1 child. I'm an only child myself and it's so lonely. How can I get over the fear of telling them if i become pregnant and not be so worried about their opinions? I know we are both adults, married, and are steady financially so their opinions shouldn't matter but for some reason I'm so worried they'll be disappointed and angry. They weren't the happiest when I got pregnant with my daughter but are so grateful for her now.