Quick vent sess.. 4 month old and dad

So my baby is about 4 months old. I went back to work at 6weeks as a nurse full time so i work 12 hour shifts a day and each week i either work 3 days or 4days. And while I’m at work my son’s father has him or my son’s father goes by family and they help him out. Cool... as soon as i get off, I got get him. On my off days, i have him. Which i love my baby to death and want to spend as much time with him as possible because i work. But when i need to do things like get my hair done or need to go run and do things around town and ask him to keep while i go do things, he guilts trip me and says things like “why can’t i take him with me or maybe i shouldn’t do that thing today. I have no family help close by. My mom stays and hour and a half a way. My grandma stays in the same town but she had stage 4 cancer and gets chemo and radiation being a nurse I know what that does to her and even if she says she doesn’t mind, i don’t want to put that on her.... just frustrated idk. I am wrong....