SO sibling

Me and my SO have been together 5 years, married for 2. My SO has a sibling that is my age. We were pretty close at one point and always goof around every time we're together. But for the past 3 years or so, when I'm around them, I'm very attracted to them. Physically and sexually. When we're away from each other, they aren't even a thought, only when they're in my presence. I try to be open with my SO because we have a honest marriage and relationship and I don't like to hold things back, but it's to a point where I'm passive aggressive towards my SO because in my mind "I can't have the sibling". I feel horrible about everything and if I could change my feelings I would but it's not something I can. My SO is hurt and doesn't understand where I'm coming from bc they have never felt that way about another person. Everyone may not approve but can anyone at least understand that I can feel this way? Or if you're in the same situation? What did or would you do?