Newlywed/expectant parent troubles

My husband has a daughter (4 yo) from a previous marriage and she lives 4 hours from us. They have custody papers in their divorce decree but they've never been followed, and the mother moved after the decree was established. His ex wife gives his daughter to his mother for a weekend about once a month and he's expected to travel the 1.5 hours to spend his weekend there at his mom's too. Problem with this now is he has a wife and baby on the way and cannot be told on Thursday morning that he needs to spend the weekend that far away bc it's convenient for the child's mother. His little girl is sweet and precious and loves me and is super excited for a baby brother. But truth be told, we hardly see her. We have gone when told (last minute) since being together and spent the weekend with her. We have decided today we need to get an attorney ad establish a new custody arrangement. For the mean time, my husband (lied) and said he has to work and can't do the last minute weekend stays anymore. Are we in the wrong for standing our ground? Should we just suck it up and drop our plans to spend our weekend there? She literally doesn't even tell us when she's taking her to his mom. Advice please.