Slow let down?

Okay, so I think I figured it out. My LO will nurse on one breast and then towards the end of his feeding, he starts to get super pissed. I am pretty sure it's because my let down/supply slows down. I will put him on the other breast and that seems to work for him... But what do I do?? Do I need to work on increasing my supply to make sure he gets enough? I just know he is growing and not having enough is terrifying me.