HELP Decreasing breastmilk supply

I had my first baby three weeks ago.. and my milk supply came in within 3 days after giving birth.. I was pumping about 5 oz each breast almost every pumping session.. ( I primarily pump, my son won’t latch to my breast correctly, but I do try to encourage him everyday) since he doesn’t feed directly from my breasts, I pump.. but recently this past week I’ve noticed my supply has gone down by a lot.. I’m only pumping minimal 3 oz max is 4 1/2 oz with both breasts.. what do I do? Is there any tips or advice to help me regain my milk supply I was previously at? I’m worried I won’t be able to make enough milk for my son ..