Social pressure to have a baby

My SO and I have both decided that we probably will not have children. We both like kids and have many in both of our families, but our long term plans would be difficult to complete with children (i.e. traveling, starting up our own farmland, having private businesses, sex/napping whenever and wherever we want, etc). We are both very happy with our decision, but experience some negative feelings from our friends and family. Everyone always calls us selfish or tells us that we're wasting our lives and relationship by not having children, but I just don't see it that way. We love each other and love being around each other, why does not wanting babies make us criminals?