What would you do.. Blended family troubles. Long post

Ki
My daughter is from a previous relationship. One that I left and moved away from without much of a work to her dad. I've always felt guilty for how I left and made every effort to make sure he is a big part of my daughters life. 
He lives about 5 hours from me and so he sees our daughter 4-5 times a year for a week at a time. 
My fiancé has always harassed me for bending over backward to make sure she gets to see her dad. He says that they(her dad and his family) take advantage of it (which they do) and I should just say no when they ask to see her. 
Fast forward 5 years. We had made a plan that he was going to take her for 2 weeks in July and 2 weeks in August. He took her in July and I had assumed (nothing had been talked about) that he wasn't able to take her in August, so we carried on with plans and life. He messaged me yesterday and wanted his two weeks. I told him no because she needed time to adjust before school starts so a week is the most he could have her for, but because we had plans this weekend he couldn't take her until Saturday evening and she would have to miss her brothers birthday for her to go. I wanted to have her make the decision because it is her relationship with her dad and her relationship with her brother, and it isn't really my call to make. 
My fiancé is pissed at me for even considering her missing her brothers birthday and thinks that I should tell her dad no, and that they can wait until Christmas to see her again. He said it's unfair to her brother (which it is) and that, that should be enough to keep her home. 
I feel like he is putting our sons wants and needs above our daughters wants and needs. Am I wrong to allow her to make this decision? 
What would you do in this situation? Btw my daughter is 7 and my son is turning 4