Is it wrong to have a crush on a mentally handicapped person?

So this guy I work with is mentally handicapped. He’s fully aware and functional, he just has a speech impediment and can’t read big words. The barrier is there but it’s small. I just feel weird about it a little bit because some people may call it “taking advantage” and I’m just wondering what your thoughts and opinions are.