Am I wrong? Opinions please.

So my son is one years old and his father has met him one time when he was maybe 6 months or so and when he did meet him my son cried and fussed wanting me the whole time and after his father left that was the last time he has made an effort to see him. Now my sons father is texting telling me he wants my son to go to Virginia for a month with him. I said no, due to my son not knowing him, trusting him, or being comfortable around him. He is telling me I’m being stingy and I’m wrong for not allowing him to go. I personally am thinking of my sons well being and comfort, but he doesn’t think I am. So my question for you all is do you think I’m wrong? / side note; my son is with me 24/7 never leaves my side and cries bloody murder when I’m not around even with people he’s comfortable with and knows, so leaving me to somewhere brand new with people he doesn’t know I think will also stress him out, so that was apart of my reasoning for saying no. His reasoning is that he’s his “dad” and will just have to deal with it and get comfortable, also said “you act like he won’t be fine after I buy him a kids meal or something”. So again give me your opinion on if you think I am wrong or if he’s wrong or right. Thank you!!