Child pick up!

My soon to be ex husband (after the baby is born)...is always late picking up our two kids on our verbal agreement for visitation. Its always on his time or when he wakes up in mid afternoon. I dont feel like i should let him see them if he cant even obey by a simple time schedule. How is he going to do it when the visitation is set in court?!!! My kids nor myself do not run on nobody's time especially not his. Does anybody know if i have to let him see the kids if he doesnt abide with what we have set already?