glucose test?

Can I opt out of the glucose test?
I had it done at 16 weeks, I failed the first time then passed the second. 
Im 22 weeks 6 days and I switched to a new OB and they want me to retake the test at 24 weeks. But I feel if I didn't have it before the damn drink is gonna give me diabetes on it's own 🙄