Should I take him out of the friend zone?

so theres this guy ive known for probably six years. back in grade 8 we had a 'relationship' which consisted of no more than holding hands. anyways i got over him pretty quick when it ended. we've stayed really good friends throughout all these years and are now graduating in a month. he's had a crush on me for all this time and has expressed that quite a few times. ive friendzoned him though because im just not attracted to him physically. Im debating giving him a chance though but not sure if i should. he asked me to prom and i said yes and we've gone on 2 unofficial dates now since hes asked me. hes honestly the nicest guy ive ever met hes probably my closest guy friend and such a gentleman but i cant get over the fact that im not attracted to him physically/sexually. should that make a difference? am i being mean or rude because thats what's stopping me? hes not an unattractive guy just average but idk.