Nonconsented?

I met a guy and I told him I did not want to have sex. He convinced me to hang out last night and I told him again. I also told my friend and his friend that I was not into it. His friend brought him condoms and he ended up getting me to have sex with him. He says I can't be mad because I didn't tell him no verbally... yet I pushed him away and told him I did not want to before anything went down. Is he right or do I have a right to be upset?