Is this wrong?

Straight to the point, I want to start a poliamorous relationship with my boyfriend and his brother. I love my boyfriend to death, but I'm getting feelings for his brother as well. Both of them love the idea. But I just need to know how icky it is. 
PS it is not long term, I plan to marry my BF