Am I wasting my life with a gay man

Hi! I've been with my SO for nearly 4 years and I absolutely adore everything about him, we're looking to buying our first house together in a few months. Everything is perfect apart from my concerns on his sexuality. When he was 17&18 he slept with a guy multiple times but never came out. Now 24 he has desires to have threesomes with myself and another man. I tried to become more adventurous and we use a strap on but I'm so worried years later he'll come out and I've wasted all this time and be heartbroken. He has 3 gay friends he sees a lot and also went on a trip away with one, he turns angry and annoyed When I've tried talking to him sooo many times over the years but just says it's only when he's turned on he thinks about it and only loves me and would never go near a man. Any one been in this situation? Am I over reacting and he's just kinky?? Any advice appreciated! :) xx