Would you go outside if your babies were sleeping?

I have a toddler and a baby under one. We have monitors including a video one. Once they go to bed, my husband and I like to go out in our backyard and spend time together whether that’s swimming in our pool or just sitting out and talking. My sil was saying that it is really irresponsible to be outside of the home while the kids are in there because something could happen. Which it could but that’s why we have the monitors to check in on things. Would you do this