Should I be worried?

I’ll be 23 weeks tomorrow and my Doctor said not to worry about counting kicks until 24 weeks when kicks become more regular, but I’ve been feeling baby pretty consistently for the past few weeks but yesterday and today I’ve hardly felt him at all. I have felt some little kicks but only a fraction of the amount as normal. I have a fetal Doppler so I checked for his heartbeat and it’s normal. I’m just worried but don’t know if this is something I should goto the er for or if the worrying isn’t helping anything? Do babies have less active days when going through a growth spurt or something?