my 5 week old is constipated 😭

My baby is constipated 😭 he's formula fed and I think this may be the problem. He's been having the same formula since birth but haven't had any problems up until this week. (I plan on changing formula after I speak to his doctor. Anyway he's started going nearly 3 days between poops and he is clearly uncomfortable as he grunts and strains and kicks his legs. Today he did a poo that would be considered hard and it was only a small-ish pallet. I've heard that giving him a little bit of cooled boiled water could help? has anyone tried this? Is it safe to give a 5 week old baby water? Help!Â