Wearing a shirt from my current college to work?

My work is pretty chill, and as long as you look nice/presentable you can pretty much wear whatever. I have avoided wearing shirts that would identify me to a certain place that I would be (such as my college). I do this so that customers (who could be anyone) wouldn't be able to potentially track me down/follow me and harm me. I asked my husband what he thought about me wearing a college shirt to work and he didn't understand my hesitation. So I guess I'm wondering if its all just in my head or if other women have thoughts along the same lines? Am I just thinking about it too much?