Late talkers?

So my son is a year and a half old and he only says a few words (mama, dada, used to say nana for food but hasn't in a while, occasionally he'll say hi or hello). His doctor says if he doesn't learn. More words by his 2nd birthday he'll need to see a speech therapist. I feel like it's somehow my fault I guess that's my mom guilt kicking in but what are some good ways to help him talk more. He knows and understands what I'm saying but just doesn't seem to have an interest in talking more. Anyone else have a late talker? Should I worry or just let him take his time when he's ready?