Baby cries and I feel bad

What do you do when you baby cries but you can’t pick the up? It breaks my heart if he cries and I can’t get to him right away. He is 3 weeks old. If Inout him down for a nap and he wakes up while I’m in the shower or going to the restroom it makes me feel horrible if I can’t get to him ASAP. I typically rush to finish and get to him but sometimes it take a few minutes and it drives me crazy. My husband can handle it better then me. Does anyone else have this feeling?