Condoms and Oil

If me and SO were to use a massage oil during foreplay, would that affect the condom at all? I know oil based lubes deteriorate latex condoms and cause them to break. We don't plan on using it as a lubricant though, just simply foreplay.