help me face my fear 😕

A while back I made a mistake and had sex  with this guy whom I thought we were just friends . A couple weeks went by and we sweapt  it under the rug.. we slowly stopped talking after that but I'm hurting on the inside ( I tried not to get emotionally attached ) . my friend guy said I should face what ever is bothering me about him and talk about it .. 
but the thing is I don't exactly know what about this situation is bothering me and if I was to say something to him what would I say?😐😕