18 month old overnights

My ex and i have a custody agreement. We live 2000 miles away from each other. He has asked for one weekend a month Saturday at 9am until Sunday at 5pm. He hasn’t come out in 6 months to see our daughter. When he does come out it’s only with his mom so she can take care of her. The two times they’ve been out they always ask for 4/5 days instead which I’ve been accommodating. I have recently read how it’s not great for toddlers to spend so much time away from their primary care giver and can creat attachment issues as they don’t understand you will be back. I’ve noticed she seems so withdrawn and almost mad at me when she’s came back in the past even she though it’s only been twice. Am I being unfair if I ask them to stick to the agreement? I’m only concerned about what it does to her psychologically. Am I being I’ve dramatic? Has anyone done this? These people are basically strangers to her now as her dad and I have been broken up since she was 10 months old. When they do visit it’s almost 2 hours away where they stay so it’s not like I can just pop in and see her. I also am not very friendly with her dad as he is the definition of a narcissist and was very emotionally abusive.