What do you do with your baby during the day?

I feel a little ridiculous asking this question but I feel unsure about what to do with my three week old when he's awake during the day.  He doesn't seem to want to be held but I feel guilty just putting him in the swing or on the floor to entertain himself.  I do exclusively breast-feed so try to remind  myself that I spend lots of quality time with him in that sense but I guess I'm just feeling a little bit like a failure today. 😔