Switching doctors 21 weeks

I want to switch doctors to the one who did my last ultrasound. He’s worked in the county for 30 years and is amazing at his job. He doesn’t want me to switch because a lot of patients switch to him and he doesn’t want to offend the doctors. I’m so frustrated! My current primary Dr and I barely have a relationship and I don’t feel as comfortable as him. In the beginning of this pregnancy he told me not to hold on hope because I was spitting and "most women lose their first pregnancies." He didn't even bother to do an ultrasound at 12 weeks which we scheduled so I missed the window for specific genetic testing.  I feel like I’m getting sucked into what many other women do, even though I went into this with clear ideas about what I wanted. A midwife, a home birth if possible, and a good relationship with my care providers. I know this is only my first baby but this isn’t turning out how I’d hoped and now I’m halfway through and feel like its too late to go anywhere else. 

Has anyone ever switched to a different office this late in pregnancy?