Would this be an issue for you?

OK here's a long story. I have these people that took me in as their own family. The mom is 60 the daughter is my age and the son is a year younger than me. My own mother gave me up and my grandmother raised me. I met this family when I was 14. I call this lady my mom and my kids call her grandma. I am now 30. When I met my husband I told my husband I would never give up this family for anyone including him. They have always been there and treated beyond well. My husband gets mad because I want to take my kids to see them they moved 4 hours away. She comes and sees us every three months and stays with my kids my husband and i. I dated the son when I was a teenager and he says he doesn't want me around him. The son is getting married and we have no intentions of doing anything together. I am an adult, I love my husband and It was 13 years ago when me and her son ended the relationship. There is nothing between us. I have never gave my husband distrust in me this way either. My husband still hangs out with women he had previous relationships with and I have no problem with it for the simple fact we are adults and I feel our marriage is stronger and than an old fling. What do you all think? Should I still go and see them this spring break or just not go?