Breast milk not drying up?

I'm 5 weeks pp and baby has been on formula since birth. I have been using breast pads from the get go but for some reason I'm still producing milk?
The only time I notice it is when I take my bra off. The breast pads are now bone dry so I keep thinking that's it, but as soon as the bra comes off they start leaking again. Any ideas on why this is happening, will it stop?