I am torn on what to do with my kids and family..

I have a son, who is 6. I have a daughter who is 10 months.. my family (dad, sister, stepmom, tia) basically disowned me when I got with my daughters father because they didn’t like him.. but I thought whatever they didn’t like my sons father at first either..

well I’ve been estranged from them for about 3 years.. only seeing my dad when he wants to get my son.. and then my son spends the weekend at his house.. seeing my sister and everyone who wants nothing to do with my daughter or me..

this past weekend my grandparents came to visit me and my daughter (my dads parents) and they told me I need to come back to the family.. which I’ve tried a million and one times.. they say they aren’t ready.. that by choosing my husband I am not family anymore.. my grandparents agree it’s ridiculous of them to act like this..

my son comes back with new toys and clothes and all these fun stories.. something my daughter will never have.. I think I want to tell them straight up if they don’t want my daughter and me around, my son won’t be allowed over there anymore! Shit, my exes family has seen my daughter and I more than my own blood! My exes sister takes my daughter when she takes my son! His mother does too occasionally! And my own family won’t even meet her!! My sister hasn’t, my tia hasn’t, no one but my dad and grandparents a couple times in passing.. besides when my grandparents came to my home

Should I let my son go somewhere his sister and mother are not welcomed? My husband has no family, so it’s not like she is getting love from grandparents anywhere else..

Edit; thank you for the comments.. every single one made my cry.. it’s hard, I feel like I’m taking away from my son.. but you are all right!! I’m going to calmly tell my father he can’t have my son anymore.. I know he will say I’m being crazy and bipolar (he always thought I was bipolar but I’ve seen many psychologists who say I’m not)