Is it alright to take pictures of someone without permission?

I'm curious, how do you ladies feels about people taking pictures of you without your permission, whether it's at home, school, or work? About ten minutes I was talking with my friend and a guy nearby took a picture of us without asking. I was livid. I might have let him take a picture had he asked (probably not because he's an ass, lol), but taking picture without permission is wrong in my opinion. He got mad at me for reaching for his phone to delete it.

So do you think it's okay to take pictures of someone without their permission? And how do you react when someone does it? What do you think is appropriate reaction?