SO and Friends

So my SO has this friend (female) that we work with and she has in her past slept with many of the other guys we work with and I don't feel comfortable with my SO and her around each other him and I keep getting in fights about it but shouldn't my comfortableness be a semi priority?? I'm not asking them to not be friends I've never said that but I have told him that I feel uncomfortable with it but I'm willing to deal with it since they're friends but it hurts