When is the right time to let his father keep him by himself?

My son turns 3 in August and after one year of his dad not being in his life, he’s back. We have very slowly put the past behind us, and worked on building trust and things like that, we have been in contact again since this past March. Next week will be their third visit (with me there.) but they have FaceTimed quite a bit. I just wanted my son to get to know him before we dive right into the him keeping him by himself, because honestly my son has never really known him, and I’ve never had to “share” him before so this is new to everybody and I’m so scared over how this will go, but I was scared over the first two visits (the last visit he got to see grandpa for the first time in two years and automatically knew who he was) the visits went amazing and you would’ve never even thought that he spent a second away from any of them let alone years.

Anyways I was planning on if this visit goes well, letting him keep our son for a few hours (or the whole day depending) on his birthday so they can have a little party for him and I usually don’t plan our event on his actual birthday anyways (usually I do his party the weekend after) so it wouldn’t really be taking anything away from me at all. I’m just so nervous as to how it will go. But I have to keep reminding myself that, that’s his parent too and he needs his Dad as much as he needs me.😬

We’re also super young so we still have A LOT of learning to do when it comes to this type of stuff. And due to my dad being an absentee father I have bad trust issues when it comes to stuff like that, especially my kid. That’s why it worries me so much.