Court question

Me and my sons father both share Legal joint custody over our son. I take my son to his appointments and make all of his appointments. I let the dad know after my son sees his doctor everything that’s going on. Now he’s threatening to take me back to court because I don’t know let him know the day of. And being that we’re in a pandemic only 1 parent is allowed up with the child. Would I lose rights to my son because of this appointment situation?