Am I being selfish? Update

So I failed my 1 hour glucose test, it was 168. They want me to come in for a 3 hour test which they've told me I can't leave for the entire test and can't return to work in between blood drawings (my office is practically next door to the doctor.) I really don't want to miss work for 3 hours...is it selfish of me to refuse the 3 hour test? I mean, from what I've read they will pretty much just put me on a strict diet and keep an eye on me if I did take and fail the 3hr. Couldn't I just go ahead and take all the precautionaries as if I did? Has anyone else ever refused a test like this during pregnancy?

Update: So glad I didn't get too shamed into not speaking to my doctor about the test. She didn't really seem concerned about my levels or the health of me or my child as we've been perfectly healthy thus far. She said she has some women to refuse the entire screening all together for personal views? Either way she didn't press the issue, only saying the test was preferred but not mandatory and gave me the option she has offered to plenty of other working moms; to take the test after work at the hospital which is open later...So it worked out perfectly!