too far too fast?

So my boyfriend and I have known each other for 4 years and have always had underlying feelings for each other. We finally started dating 3 weeks ago. I know this will be judged upon, but he rubbed my clit on our bus after just two days of being together. We have gone as far as we could, with oral and everything, but we have both agreed to not have sex until we are older and ready. Yesterday, he was really into t and asked if we could have sex and I said no. He asked again and I said no, and he stopped asking. Later, I was grinding against him but with panties on and he asked if I could take them off, and I said no. He slightly moved them after I said no, and I sat back down and told him that I said no. He realized what he had done and he kept apologizng to me the whole day and was disappointed in himself. Do you think we are going too far too soon or does it just matter on how comfortable we are with each other?