Special needs people

There's a man in his 30s at my office who has asbergers syndrome and maybe something else. He constantly says and does inappropriate things, including make comments about women's breasts and behinds. He's got a particular fascination with pregnant women. I am currently pregnant and often find him staring at me--like really staring. To me it feels peverted, especially given his history with sexual comments. He also tries to stand or sit around me whenever he can. I always move.

 

Other notes, he never stops talking, has no filter, says whatever is on his mind; it usually makes no sense at all. He makes a lot of arrogant comments like "I'm so good at ____, ha-ha!" He even talks when no one is around or paying attention!

It's very annoying and makes him rather dislikable. I'm not the only one who feels this way. Lately I have gotten quite sick of feeling harassed and complained. I was told he "can't help it" and to let it go. 

So my question is this: should I let it go? Can he truly not help it? Why is having a disorder an excuse? To me it seems like if you're an adult who drives and functions around other people you should be held to the same standards as everyone else. At LEAST with the sexual shit. But that's just my opinion.