I NEED ADVICE

Okay so I was hanging out with a guy today at my house. When he was about to leave he kissed me. (It was my first kiss.) After that it was really awkward and I just wasn't feelin it anymore because I had liked him before. So I told him that we need to just be friends. My brother HATES him and I don't know what to do. Please someone help me. I don't know what to do. Did I do the right thing and reject him or should I give him a chance?