Yeast with no infection?

Okay so I always have vaginal yeast (and sometimes itching but that depends on the underwear I wore that day) but A while ago I got a Pap smear and nothing was wrong. The lady said “you have some yeast” but the result came back as no infections of any kind. Every time a penetrate when I masturbate I get this nasty toilet paper like discharge on my toys or on my fingers. That’s the only time I really see the discharge but the usual discharge on my underwear seems normal. I dont know what is going on or how to get rid of it but It’s making me insanely insecure about having sex because I don’t want anyone to think I’m dirty. Cus it looks gross. Is it normal? Should I go back to the doctor again? Like i dont know what to do.