Do you think this is rude?

When I eat at my office, I put a closed sign up. I like to sit at my desk because I like to take care of personal things on my computer. Or maybe sometimes catch up on Netflix. I have a lot of windows in my office. When people see me there, obviously a sign is on the locked door, they will start knocking on the door. Sometimes, it's not even my personal time but for meetings or classes I have to take for my job. It just seems strange to me that literally any person that walks up will knock when they see me. I know they see the sign. Would you knock on a door during someone's lunch?