Being worried

So I'm talking to my family again and my grandma asked me to come out for mother's day and bring my son I told my boyfriend this and hes worried about them not bringing me back but I told him that he doesn't have to worry and they are allowed to see our son too (no hes not keeping me from my family or anything) he doesn't want to go cause we had a fight and I stayed with my grandparents for a bit with our son