Can you let your baby sleep too much?

My son, who is almost 4 months, sleeps through the night every night and then as long as I’m laying with him for naps in the day(we co sleep)- will seemingly sleep indefinitely.

I just keep my boob out so he eats as he pleases and while he’s still mostly asleep and I have diapers by the bed and I keep him dry by popping a new one on him every once in awhile.

Lately I have been so exhausted and have been staying in bed and I just feel guilty because I wonder if he’s getting too much sleep just because I want to nap... thoughts?

I still make sure to get in lots of interaction with him, reading to him, having our pretend conversations, and we do tummy time everyday.