Stay or let it go...

We started out as friends with an open relationship to see others outside of us being together as long as we were respectful. Now both of us have feelings. Recently I feel that he may be shutting down or pulling away, what is your opinion? I think it's because we had an agreement and he's not happy with it. But I don't think its fair to say you have feelings and pull away. If he wants a relationship then he should say so right, so that whatever goes on outside of us two can stop. Help...