Advice?

I am looking for some advice. Please don't judge too harshly because I already feel terrible about lying.

My boyfriend and I have been together 6 months. We were talking last week, and somehow we got on the topic of when we lost our virginities, which led to him asking if he'd ever met anyone I've slept with. I said no, honestly thinking that was the truth. He then said that he'd thought maybe I'd slept with my old co-worker. I had forgotten they had met back in February or March at some random event. I don't know why he mentioned this guy specifically. We had been friends a long time and have some pictures together on fb, but we only slept together twice and never dated. Anyway, I felt panicked and awkward and I said no and moved on.

Should come clean and tell him about it? I don't want to lie and have him find out later and have it be a bigger deal than it is. On the other hand, I feel like it shouldn't matter to him at all. I don't know how many people he's slept with or who they are, and I don't want to. I think that bringing up the past would moreso ease my guilt about lying than do any real good with him knowing what happened with this guy. Thoughts?