Using soap down there

Is it bad to wash with soap down there? I once had to go to doctor because my inner lips got very itchy and swollen. I had been using a soap that was supposedly made to wash myself down there. Apparently that soap caused bacteria in me. The doctor recommended to use dove soap the regular unscented and ever since then it's been working fine. But is it bad to use soap?