caring for baby advice needed!!

Okay so my mother in law watches my son for probably 6 hours every day and my husband watches him for 4 hours every day, and it seems usually when he's with him he's fussy(maybe because my son can feel his bad mood?instead of my husband trying to comfort him in a loving way he gets stressed out and doesn't do what he needs to console him) but anyways I feel like maybe I just need to risk my sleep even more and just spend more time with my newborn because he seems calm and happy with me most of the time... So with that being said should i just say no to the help and spend more time with my baby since it could be causing my baby stress? But I also think about how my husband should just get use to it and be able to handle it, but maybe he's just too immature and doesn't have compassion on our baby?