BF's number

The number of sexual partners my BF has had bothers the hell out of me. I can't stop thinking about it and it frustrates me. He said that he never was looking for sex but the number is way too high for him to say that. It's because I lost my virginity to him that I feel so bad about how I've shared my somebody with a number of other people. 
Is there anyone else out there whose dating someone and is in a close or similar situation that can give my advice on how to calm down about this?