Is it okay if you don't say no ?

When I was eighteen I went to a beach house party with about twenty close work friends for a weekend and I decided to hook up with a guy I had been talking to and hours later my really close guy friend told me he knew what I did and hated me and that I'm disgusting and that I broke his heart and all this Terrible shit and I was drunk and upset that this arguing was public and in the middle of what should have been a great weekend ... I calmed him down and we fell asleep on the couch at around 4am and I woke up to him kissing me and I said hey um nope what's going on and he said oh you can fuck him but not me after all I've done for you ( he used to buy me drink and food at all our work events and parties we went to ) and I said it's not like that I do like you so he said okay good and proceeded to take my bathing suit bottoms and T-shirt off and doing stuff but I didn't want to say no and make him more mad ... so I just laid there staring at the ceiling until it was done... he rolled over and went to sleep and I got up and went pee .. took three shots of vodka in the bottle of the counter smoked a cig and sat up all night ... it didn't feel right but I didn't say no ... I haven't thought about it much at all until my close friend was telling me about her getting assaulted a few months back... and I just felt sick .. I haven't told anyone I haven't talk to the guy since I don't work there anymore .. but I just remember how awkward it felt and how uncomfortable I was ... I don't know what it was but for some reason it's bothered me since it happened even tho it was three years ago and I'm now twenty one it still freaks me out that I didn't say no and I didn't stop him and I acted like nothing was wrong the whole weekend .. I guess I just drank the feelings out of my head and when he came around and acted like nothing happened at work I just let it go ... I don't know why it's bothering me now ..... ugh what's wrong with me ?