Question about legal advice with vacations

So my sons grandpa wants to take us on a vacation to Florida and I'm not sure how that works though because my sons dad and I are not together and we're never married. He also very rarely sees our son, it's been almost a month since he's seen/asked about him. He texted me the other day asking if we were free because he wanted to see him "all the days" whatever that means, I said he's welcome to come see him whenever but I have not heard from him since. Would it be okay to take my son on vacation? Or is that like not okay because my sons father and I aren't married I'm just not sure how that works.  We have never been to court, or even spoke of court!