Am I crazy.. or is this on him?

It’s an extremely backwards situation. My sons father and I are trying to work on things but we do not live together. We have a 1.5 year old and I am currently due in June. He also has an almost 11 year old from his previous relationship. He doesn’t see his oldest much so when he does get him he gets VERY weird if I’m around and filled with anxiety. He lets his son disrespect me and his son turns very whiney and unhappy when my son and I are around. He shows many signs of anger towards my one year old. Though we are barley ever around him. My sons father won’t even invite us to do things together anymore. But he expects us to move in together soon?! He just got a new job where he is working a lot more and gets to pick his day off. He made sure to get Saturday off so he can get his oldest. Needless to say we don’t see him on Saturday’s. Well now that he’s working so much he expects to come over after work 1-2 nights a week (after 8 o’clock) when my son is already sleeping. So he stays and my son got up at 7:15 this morning and his dads still in there sleeping. Probably will be until he has to leave to get his oldest. Am I wrong to be upset that he makes is a priority to take a full day off for one kid but can’t even wake up a little early to spend and hour with the other?!