breast pump cleaning

I haven't sterilized pump parts at all, I've been rinsing them out with water, not even soap. I honestly didn't know it needed cleaned since I'm the only user... Is the milk I've pumped and froze bad and needs thrown away?