Face breaking out when I wash it?

Does anyone know why my face could be breaking out when I wash it? If I don’t wash my face I barely get any acne, but as soon as I wash my face for a few days I start to break out. I’m not using a harsh cleanser, if anything it’s very close to being natural. My face is generally clear if I don’t wash it although I’ll have a bunch of blackheads on my nose, my cheek part right next to the nose and my chin. Those areas started clearing up now that I washed and exfoliated them but the rest of my face started breaking out. I don’t understand this? Help please!