wtf? am I right to be upset?

C
So y boyfriend and I have a 5 month old together and he has two kids from a previous relationship. I've spend the summer taking care of them while he's at work as well as our baby daughter. We've been fighting a lot lately about stupid things... I guess neither of us feels very validated or appreciated these days. I've asked him repeatedly to try and be more thoughtful and he just doesn't seem to be able to. This morning I'm feeding our baby and he asks me if, in a few weeks when his kids are with their mom, if I could go by my moms for the weekend (about an hour away) so he can rest.
I'm so put off by this but also think maybe I'm overreacting because I've been a little short tempered lately. Is this not kinda fucked? I don't ever get any weekends off to rest. He's never up with the baby in the night and on the rare morning he will get up with her. I just feel like this is kinda bullshit