I need to vent!!!

Little background. I've known my husband since he was 12. I have 2 kids from a previous marriage and he has 2 kids from a previous marriage. We love and adore our kids. Our families all get along and my husbands family is like my family. We both work (I work from home) and we aren't on any state assistance. My husband and I want one child together. Well my husband wants 2 more but I think one is perfect unless God blesses us with twins. Here's my vent. Our families keep telling us we better not have anymore. They have no clue we've been TTC for 7 months. We never ask anyone to watch our kids unless someone asks if the kids can sleepover. We don't borrow money from anyone. We support our family (kids and ourselves). Should I let their opinion bother me? I want them happy when and if we finally get our BFP. Please no rude comments.