those of you who have estrogen only pills!

do you think just not taking any pill will make a difference? like why exactly do they put those in there with the two inactive pills and then just have estrogen only? I always forget to take them and I was wondering if it really made a difference since its "inactive" week