How do you feel about your SO going out with opposite sex friends without you?

To make a long story short my SO wants to go out with his female coworkers without me. He has a past history of flirting with them and even cheating with them. I'm not trying to control him, but it hurts my feelings. I know I get jealous but I can't help it. I told him that I don't mind if he goes out, but I want to go. His excuse for not letting me go is that I will get mad when he talks to his female friends. He constantly tells me I'm the only one who has problems with this and I'm nuts and I need to seek help.