Guilty sex conscience

I expressed to my significant other how I would like to go to a sex party. Not necessarily to take part but experience and see. He asked me if I would have sex with someone else or be ok to see watch him.

I was hesitant to answer. He said imagine we were in a very open minded relationship. So I said - if it turns u on and you’re ok with it I might but idk how I would react in that situation. I can fantasize about it but I don’t believe it’s something I would actually do-

When I asked for his answer in return I learned he was bothered by my answer and he just kept saying he would never want to do any of that.

I feel his perspective about me changed. I feel and I told him he tricked me into answering because he said if the situation is we were both open minded. And kept saying that each time I tried to get out of answering but he wouldn’t budge when it was his turn to express to me his fantasy.

Am I wrong to feel concerned? I feel like I did something wrong and have no clue on how to fix it.