Deployment patches!!

We are a military family. My husband has been in for 4 years & served. And is in for another 4 years. He's going until he retires. After his first deployment, he doesn't wear his deployment patch at all now. I guess it's not mandatory. He says he doesn't like wearing cause sometimes when we go into town off the army base he says ppl treat him differently. Whether he's in uniform or not. He doesn't always wear his uniform unless he just got off work and don't have time to change his clothes. But he feels like ppl are more respectful when he wears it and it bothers him. He doesn't enjoy killing ppl for a living. He always walks around at the office without his deployment patch. Anyone else's husband bothered by showing their deployment patch???