Sugar and general diet during IVF

Does anyone cut anything out food wise during fertility treatment? I eat healthy, excerise, walk to and from work everyday (4 miles), but I'm such a sugar addict. I just can't give it up no matter how much I try. I've already given up coffee and alcohol for treatment. After 2 transfer failures, I'm just pulling at straws and wondering if I need to cut sugar. But it just feels like another added stress to give it up. Which I feel like I don't need. Has anyone changed their diet or cut anything out, especially sugar, and had success after failures? Or do most people think that if the embryo is good, that stuff doesn't matter as much?