Starting to feel like a failure☹

My baby is 3 weeks old today! I've been breastfeeding but haven't been pumping as much as I should, now my supply has gone way down.

Last week I was getting about 3oz per breast each pumping session, I just finished one and barely got .50oz total😭

My mom has been visiting from out of state this week, and she's made me feel like I need to constantly be with her so I haven't been pumping as much (maybe once a day) and I haven't had chance to nap. Now I feel like I'm failing my baby by ruining my supply for her. My mom is leaving today, and my husband told me to just pump more but I'm not sure if that will fix it again?