Hooking up with someone's brother

I went to a party over Halloween weekend and started talking to a guy I met there. I ended up going home with him. Now I knew he was the brother of one of my teammates and that he was also a 5th year senior (I'm a freshman) The next morning he walked me back to my dorm and I got his number. I figured that would be the end of it. Well that morning at practice one of my friends asked if I hooked up with him because she saw me leave the party with him. I said yes and she basically told me that it was a bad idea since he's (we'll call his sister Sally) Sally's brother. I listened to what her thoughts were and kinda blew it off. That night we had team movie night, one of my other teammates pulled me aside to a different room during the movie and got mad and started basically whisper yelling at me for hooking up with Sally's brother and how it wasn't cool to be sleeping with teammates siblings. Afterwards I got a little upset but kept it to myself and that was that. About a week or 2 later the brother texted me asking if I wanted to come over. I said yes and we watched a movie, hooked up and just talked. We've been doing this since then and I know some of the team knows, especially more on the guys team,I don't know if his sister knows but I have a feeling she might. I don't see what is happening as wrong, neither do most other people I've talked to. But my teammates seem to see it differently. What are your thoughts on the situation?