Number of partners

My boyfriend and I have been together for a while now, and he's going to be moving into my apartment with me. Last night we somehow got on the topic of how many partners we've slept with. Now, my number is under double digits, but I'm still a little self conscious about it. So I was hesitant about telling him for fear of him judging my past. But when he told me that his number was probably higher than 15, I kind of had an internal freak out. That's a lot of girls to be sleeping with. And I mean, we love each other and our pasts are just that, the past. But it still bothers me that he's slept with so many women. Am I wrong to be a bit upset about this?