Is it wrong ?

Deb
The father of my child and I do not get along what so ever, he wants nothing to do with me or the baby at all; left me when I was 2 months pregnant. However his sister and I been talking about bringing the baby to Puerto Rico to visit his great grandparents: just me, the baby and my baby fathers sister. Now she says she doesn't care how he reacts because she herself says he isn't there for the baby and doesn't make the decisions. My question to everyone is....
Am I doing the right thing for me to go to PR with his sister and the baby and not the father???
Again he doesn't want anything to do with me or the baby, has a new girlfriend & can careless about us but my sons aunt and grandmother (his family) been sticking around and very supportive.