Am I wrong?

I'm a stay at home mom, not because I necessarily want to, but because I have health issues that makes it really hard for me to work. My SO works his ass off to provide for our little family. Long days and hardly ever a day off. Our baby is 15 months old and on mother's day morning I asked him if he could go up with our baby so I can sleep in . I have NEVER sleeped in, not even once. He got mad at me and said "No! I need to sleep! I have to work all day while you can take naps whenever you want to. Also you can't be fired from your job". I get his point but I still feel a bit hurt and sad. Like can't I get just ONE day?