Sugar Pills- Yes or No?

I normally don't take the seven sugar pills that come in my birth control pack. Is that a bad thing? I take my pills at the same time every day and have never missed a day. Do you have to take the sugar pills?