Allegra harmful? Help please!

I asked my OB if I was ok to continue taking allergy medicine once pregnant and he said anything over the counter is fine as long as it's not a decongestant. I've been taking Allegra daily and just found out I'm 4 weeks. I checked Google to make sure and read that Allegra is the only one you shouldn't take. Help! Is it possible that I did any damage or is it too early to have affected the baby? I will for sure switch to Clairitin or Zyrtec but my nerves... 😫