Trying not get upset

So I am currently in a custody battle with my ex-husbands mother (long story short she promised temporary help while we were going through our divorce, and we gave her joint custody with her wife so my son would have health insurance since my ex had quit his job and we had no insurance for him and now they're trying to keep him forever)
Anyways, I have the ability to visit my 3yo son whenever I am in town (I live 2 hours away with my husband on an Army base) so I try and get down there on the weekends at the very least. Generally it's between 2 and 4 days and then I come home. Well, my husband comes down with me most of the time when he's off work, but last night we had a bit of a disagreement. He said that he wants to spend his days off at home. So I told him he didn't have to go with me, I would go by myself. And he said he doesn't want to spend his days off at home by himself, that he wants me there too. I understand that he's my son and not his (even though we are married so technically he kind of is now) and the bitch ex-MIL won't even let him see my son, so he stays with my dad during my daily 2hr visits, but I feel like he should understand that I want to/need to be down there with my son. I could understand him not wanting to go if it was just for shits and giggles, but it's not. 
I'm trying to understand and not be upset but it's really hard for me. I feel like he should be more supportive of me trying to spend as much time with my son as possible. It doesn't help that I'm 23wks pregnant and an emotional wreck either :/ 
I guess what I want to know is if I'm being too sensitive and irrational in thinking that he should be a little more understanding and supportive instead of making me feel like I'm doing something wrong or hurtful by trying to see my son?