Unprotected sex

I have been wanting to have sex with my boyfriend without a condom but I am always worried. I am on a combined pill birth control and have never missed a day, however I rarely take it at the same time, but I have heard it is not important for combined pills to be taken at the same time everyday, as long as it is taken you will be protected. Should I have unprotected sex or not risk it?