Not trying, but not preventing

My husband and I have been together for ~5 years and we've been off BC for the last 2. People assume we're trying, but we're actually using the natural method to not have sex during ovulation. We would be ok if it happened but are not actively trying. I honestly think some people think we're "having problems" but we're not. Is this a common stage or have others recieved confused conversations from friends/family trying to explain this? (p.s. We're in our early 30s, so still young!)