Do i dump him?

So a long time ago my current boy friend and I were just talking (not putting the label on the relationship quite yet) this guy Elton who used to like me one day tried to kiss and I backed up and said im talking with someone can you leave me alone.... Now present day Daniel (boyfriend) we been together now for 3 months and he brings this up all the time like did you guys kiss? Or asks me other questions which didn't happen and I always tell him no and he is always telling me he wants something serious and in a serious relationship u have to have trust which I trust him just for some odd reason he always keeps bringing. Up that topic and everyone time I say no he says will so and so said this and that and he keeps bringing it up and doesn't trust me and today it got serious and I don't know whether to leave him if he doesn't trust me or if he is just being insecure about the subject and just wait for him to apologize and let it slide or do I dump him for the lack of trust?