Freedom of speech vs These hands are free too

Where do you draw the line between hate speech and freedom of speech? How do you define hate speech and at what point can a public conversation or campaign go from exercising their freedom of speech to being hate speech? If someone does cross the line from freedom of speech to hate speech (In a public place) should it be okay for other people to act violently (of course not killing or severely harming them though)