Advice asap!!!

I moved out of state with my two year old. I have full physical custody - no court orders. His "dad" (alcoholic, druggie, etc) called me yesterday and said he has his life together and has a place and a "possible job" and that he'd like to take my son for summers etc.....

thing is, he's met him twice. He has one 8 month old with his 19 year old gf who is pregnant again (he's 30) and now all of a sudden he wants my son who I've been raising alone to be apart of "their" family.

Advice? I really don't trust him. Would it be illegal to not send my son over summers? Ugh