giving up

I contemplating giving up ttc, it's so stressful and heartbreaking. I don't have a lot of money but I've always wanted a child and can't picture my life without children. Sometimes I feel judged because people know I drive an old car and rent a trailer home with my husband and now I'm going to spend money to conceive? The way I see it is I don't have anything fancy but the longer I wait, the less likely I'll be able to conceive. We're working towards buying a house while I get me credit score up so it's not like I'm not going to work towards better things in life.