DIRECTED AT NATIVE ENGLISH SPEAKERS

When you're talking to someone who doesn't speak English as their first language and they make minor mistakes/have an accent, do you think they need to apologize for it?

I'm not a native speaker myself and especially when I was younger I basically ended every sentence with apologies. Which I now think is kinda ridiculous as no English-speaking friend or acquaintance of mine has ever tried to learn my language, other than "hello" and "thank you". (Even if those people live in my country).