Can you fall asleep together with a condom on?

If he wears a condom, can we fall asleep together while he's inside me.
I understand that his penis will go limp when he falls asleep, and that it will come out.
But as for actual protection. Is it safe for us to fall asleep together in this way if he is wearing a condom. 
Extra note:
There would be no ejaculation in to the condom. This is how it would basically go:
He'd get hard, put a condom on, penetrate me, and we'd fall asleep together like that. 
Thought on this? Thank you :)