Pumping problems..

Ugh.. so to begin I'm a first time momma, 20 years old with a 3 week old. About 2 weeks in my supply dropped. I pumped from probably about day 3 of my son's birth.. from day one my son had trouble latching. He was born at 37 weeks and was only 4.14 at birth so his muscles just weren't capable of pulling the colostrum. All was going fairly well, I was pumping about 2oz everytime. Which at the time was enough to meet my son's demands for food. Now I've gone down to barely being able to pump a few drops.. I try to pump as often as I can, but now it seriously affects my mental health. I literally cry. I started taking fenugreek to help and that hasn't even seemed to help... I want to just give up but my hubby spent tons on our pump system and I feel terrible. I don't know what to do, im no where near meeting my son's needs for food. Of course I'm supplementing with formula but I feel even more like a failure because of that. I guess I just wanna know if any other first Time moms had this problem and if it got better or if you just gave up..