Should I be annoyed?

I have a 3 yr old and 7 month old and stay home with them except one day where I work 5 hrs. (10-3) My husband is with them until he has to go to work (3-12) and then my family members come over the hour or so that I’m not there where our schedules overlap. Working is a way for me to get out of the house a little. My 3 yr old is potty trained except for pooping. She goes in a pull up. My husband had a “traumatic” time changing her and said he couldn’t watch them anymore. I didn’t tell me family this because they know how much I need some time out of the house. So my sister and niece came over today to watch them earlier than usual but I’m embarrassed to tell them that my husband can’t because of the diaper change. So she got here at 11. He ran some errands and come home at 2 to get ready for work. I called both of them when I got off and he was still home. It’s frustrating because I have to plan out each time I work. So I called him and asked why he is still home and told him that my family takes time out of their day to come over to watch them. He said I shouldn’t tell him how to spend his time. And now I feel bad I shouldn’t have said anything to him on the phone. But if He’s just sitting at home why is my family coming over?? Am I super anxious and should I just drop it?