am I wrong?

so a little back story. my boyfriend and I have been together 4 years. we live together and have a 4 month old son. he works full time M - Th on 3rd shift, 12 hour shifts. during the day I let him sleep all day long, no interruptions except maybe a breastfeeding here or there. and even then, he goes right back to sleep. so on friday Saturday or sunday, I ask him to watch the baby so I can catch up on sleep. He does sometimes but once the baby starts crying or whimpering, I wake up. if the baby whimpers with him, I take him to a different room so he can get sleep. and it turns into a huge argument that I should sleep when baby sleeps, and he needs sleep more than I do because he works with equipment. I get that, but I need sleep too. I have other things I need to do while baby sleeps such as cook, clean, shower, eat, etc. I don't ask him to watch the baby on work days. I feel like I'm going crazy because I haven't had sleep longer than 3 hours at a time.