Over supply help

Im breastfeeding my 5 day old son every 2-3 hours, sometimes every hour and cluster feeds on demand and im still having to pump at least 2 or 3 times a day and have a stash because i have way too much milk and even after feeding i can fill up an entire bottle. My boobs are engorged almost 24/7 and im scared of getting an infection or making my supply go up even more. Does anyone have any advice? My breast get full again within a few hours and are constantly hard. Im only pumping when my breast are really painful to touch and after a feeding and i can easily get 3 ounces within a few minutes, please help😥