extra virgin olive oil?

So I've been using a soap on my face that dries my skin out like crazy. The soap is great though. Anywho I start using extra virgin olive oil to moisturize and it helps with the drying. Does anyone know if it's safe for me to do that? Will the olive oil damage my skin in anyway? Or should I switch to coconut oil instead?