a question on assault (?)

Rhi • 18, bi, lovin' :)
hey, so i'm dating a guy. we were best friends for a year, told each other everything. in november 2016 we started acting more romantic and started dating. about 4.5 months in, we were at the no shirt yes bra yes pants and panties phase  and he was rubbing my clit (through panties). we were in his room, but had to go soon and i was getting up to get my stuff together and he said "wait, come back" and i did. we started kissing again and he reached his hand down, but past my panties. we hadn't discussed that at all and i didn't feel ready so i immediately said "no. no." and he stopped and freaked out and started apologizing. this was a month or so ago. he says he didn't mean to and he's sorry and feels terrible and i believe him completely. we've gotten more intimate in other ways but i haven't been able to get back to where we were with that. i want that again, but panic every time he asks or gets close to it. i trust him with everything and love him to death but can't seem to get over it. any advice on dealing with my anxiety over this and understanding my emotions? i'm tempted to talk to my (male) therapist about it but i'm very nervous and hesitant about that...