Soap to clean vagina?

I’ve heard it’s bad to use soap to clean your vag, but when people say this are they referring to using soap on the inner labia folds or in the actual vagina hole??? Because I would never use soap in my vagina hole, but I use it on my labia minora folds. Is this okay?