Am I crazy? Wrong? (LONG POST)

I'm going to try to make this as to the point as possible, but it's a long and complicated story. And this is an odd opener, but essential to the story...I have SEVERE anxiety, so much so that my phone going off is startling and panic inducing. Only my family and friends aware of my condition and my boundaries have my number.  My husband borrowed money from a frat brother of his months ago. He had been paying him back in increments as he could, but mostly missed paying as promised. He just got his first check from his new job last Wednesday, and one of our first priorities was to clear the debt we owed. We made a financial plan around it the day prior to his payday. That Wednesday evening as he was suggesting we treat ourselves to a night out, I asked him if he had paid his friend back before we go spending money. He literally said, "something like that...". I asked him twice more, got the same response and then he changed the subject. I asked him if him avoiding the question was his way of saying he hadn't paid him back, and he said, no, he wasn't saying that either. He eventually suggested that he had paid some of it back, but he'd take care of the remainder next paycheck. I was annoyed and suspicious but eager to dismiss it because he's been promising to be more careful with money. Fast forward to yesterday. He accidentally lets it slip in conversation that he didn't pay his friend a dime because "I wanted us to actually enjoy life for a second since things have been so hard ". For the record, we've had an incredibly traumatic year but that's another story. Regardless, I was upset because he lied to me. I calmly told him I really didn't like how uneasy and hurt I felt because of him lying to me about that money. He apologized, but then reminded me of all the things he bought for our now 15th month TTC. He then said, "we've both misled each other at different points in this relationship, so.." I was confused. We had an awkward but tolerable evening afterward. And today... An unknown number calls my phone. I don't answer, because I don't answer calls from numbers I don't recognize. The number calls me back. I don't answer. The third time it called back, I answered. It was my husband's friend. I was so panicked and uncomfortable but I made light conversation until he asked how he might be able to reach my husband since "his phone isn't working" (my husband blocked his number on his phone because of the back to back calls all day). I didn't know when the last time they had spoken was, or if my husband told him to call him at my phone..I had no clue what I could/should say. I told him that I would let him know he called, we chatted some more and that was that. I decided to wait until he was off of work to address this, and I figured I needed a moment to find a way to address him non-confrontationally. He called me on his way home, let me know another couple we're friends with wanted to go to the movies when he got home but that we might not make it bc of the traffic he was in. I realized we were about to be in a space where we couldn't talk at all, so I let him know what happened. He just said ok. I asked him if they had spoken at all (no), and if he knew how his friend got my number (no). He offered to show me how to block him on my phone. I told him I didn't want to be a part of that kind of messiness, and asked that he please tell his friend that it is not okay to call me like that back to back. He said okay, then bounced off the phone as he was en route to drop off a coworker. Cue me getting around 40 text messages from my husband, telling me I'm "whack" for first telling him from the beginning to be up front with his friend about when he could or couldn't pay him so he wasn't just calling and getting ignored, for telling him to inform his friend that I literally cannot handle him calling me like that, for asking how he got my number, and for bringing this up when he had a coworker in the car (I wasn't on speakerphone, raising my voice, or asking him anything he couldn't respond to with a yes or no). He accused me of abandoning him. He then followed this with "since we're arguing..." And followed it with him informing me that we will no longer be attending one of my male friends weddings because of our single intimate meeting 8 years ago (5years before I even knew my husband existed). He goes on to reveal that he has voluntarily ended friendships with two guys he was cool with who I dated before I met him. He declared that I witnessed him "dismantle these friendships" (this was actually news to me), while he lets me have this friendship with this guy I was intimate with once (I see him maybe three times a year, and only because his daughter and my nephew are close-he's much closer with my sister). He's angry because I didn't tell him that me and this guy had sex until the beginning of this year (we've been together since Oct. 2012). I Apologized when I first told him in January for keeping that from him, and he said he forgave me, almost too quickly. I asked him if he was sure, if he was okay with the little interaction he and I do have, and if it made him angry or uncomfortable. It is clear now that since he's had time to process this and since we eloped after our year long engagement only three months later that he is actually very angry about it. I completely understand his anger. I apologized again tonight, both via text and in person when he finally decided to come home (he took a 2 hr detour), which he dismissed saying I didn't get it and that he was done sharing his feelings with me in this "relationship of grudges".
 On the one hand, I'm saddened that he has been hurting and angry about this friendship and its history that I kept from him. I actually asked a few friends (some dating, some married) in the beginning of my relationship with my husband if I should tell him about it, and they strongly advised against it, saying it was just asking for trouble, especially considering how sparingly me and this other guy interact. I ultimately went with my gut because I personally did not want to get married having anything in my past that might hurt my husband enough where it would damage the marriage.
On the other hand though, how the hell did a conversation about him lying about money turn into this?? I've never seen a script so masterfully flipped. I was told I hold grudges, and am never there for him, both of which he constantly accuses me of every time we argue. Either way, I have no clue how to proceed. Any constructive advice or criticism welcomed. Posting anonymously because this is kinda touchy...