Lube breaks condoms??

Last week me and my SO had sex. He decided to be a lil extra and use this massaging oil/lubricant. I JUST got bc a few days ago so when this whole thing happened it was before I got birth control. Anyways we used a condom and at the end there was no indications that the condom broke. I knew lube breaks condoms but my dumbass didn’t think the massage oil/lubricant can possibly break condoms. Should I be worried?