Not sure what to think?

So my fiance and I have been together for 2 years. In those 2 years, we've had some trust issues, especially after getting pregnant with our daughter. We ended up breaking up and got back together after spending some time apart. Throughout the 2 years that we've been together, he has expressed that he wants to try threesomes. Before recently, I told him no because I didn't know if I could 100% trust him if we did it. After getting back together however, and him communicating better and showing me reasons to trust him, I told him that I'm open to the idea (this was about 4 months after we got back together, so there was a lot of time we had to establish trust before we decided to try it.) Back in October, we did it for the first time ever. I liked it, I liked the connection it gave us and it gave me a chance to explore my sexuality with another woman, which I had never been able to do before meeting him. He gave me reassurance throughout the day before and the day after, but honestly I didn't need very much. Everything went great. We didn't do it until Christmas time again, just because we didn't have the money to get a hotel room (we don't have our own place) and then our car broke down and he lost his job. It's still broken down to this day, and we've been stuck at home waiting for our taxes to get it fixed. We downloaded a tinder a while ago, but he told me a couple days ago that he "didn't want to do it with me because he gets anxious about it" and I'm just confused. Like why fight for something for 2 years just to say you don't want to anymore? Why get me comfortable doing it and to a point where I can enjoy it as well, and not be so anxious with trust, just for you to say you don't want to anymore? I asked him all of this, and he just replies with "I don't know" he has an issue with getting bored with sex lives between two monogamous people, and feels like he's made for the swinging lifestyle. I never used to be, but seeing as though I was comfortable and enjoying it a lot I feel like I am too. So now, I'm just confused, and wondering if he's saying this because I told him I didn't want to do it unless he was 100% sure about us? (Before we did it the first time) advice? Sorry it's so long