Am I being a B?

Amanda • Music Therapist * Crafter * Reader * Nap Taker * Dog Lover * Daydreamer * TTC #1
So...long story. My husband owns a landscaping company. He is a super friendly guy and a great salesperson. He also does the vast majority of the labor in addition to business operations. He became friends with a customer- a cute, smart, single, female customer. No big deal. One night in December, she texts him at about 9 pm, saying "Everything ok? I haven't heard from you in a while." I thought that weird but he assured me it was a misunderstanding. A few weeks later she texts at 11:00 pm when we were about to go to bed about a concert coming to town- saying it was a band she didn't like or something. Again, he said it was a miscommunication and my feelings of insecurities were unfounded. I just felt weird. I don't text male coworkers about personal stuff late at night bc of professional  boundaries,  but also out of respect for their families.  The situation just really hit me wrong. Over the next month, their friendship progressed and he was texting with her 3-5x each week. I told him how much it bothered me, that it would not be hard for this relationship to move from appropriate to inappropriate quite easily, and how emotional affairs can be more damaging than physical ones. He continued to say nothing inappropriate was happening and started to place blame on my insecurities and my "bad attitude." I agreed that we could go to lunch with her so I could meet her. Time came up and I was sick. I literally had pneumonia. I told my husband that I didn't feel like going out, that I would prefer to stay home and not cough all over this strangers lunch, but I would let him make the final call. I was wanting him to say "no. We are canceling. You need rest." I realize this was a little passive aggressive of me and I didn't communicate my expectations with him clearly, but I felt like he clearly chose her over me. We went, I coughed and hacked, things were cordial albeit a touch awkward. WhenI expressed how I felt like I was second- that he chose lunch with a friend over his wife's health, he told me that was ridiculous. As a side note, I actually ended up sicker the next several days. This situation keeps coming up. He says I am on his case constantly about "who are you talking to. Who are you texting, where did you go today?" When in reality, those are questions I have always asked and I am curious since this is a brand new business that he is trying to get off the ground and a lot of MY faith has gone into it as well. I just want to know what's happening. 
We had a huge fight about this on Saturday. He blew up. I have never seen him in such a rage. He couldn't listen to me bc he was so angry. He told me if I didn't quit acting this way we were going to end up divorced and it would be completely my fault. He told me that he is going to run his business the way HE wants to and I had no right to know anything about it- only trust that he is going to make it work (this is the 3rd self owned business in 4 years and his 7th job). He said I had no say in who his friends were or how much time he spent with them bc it was HIS life and I am not going to control it.  All I asked was if instead of texting with her about personal stuff 5 days a week, maybe he could dial it back to 2 days a week just around when he is scheduled to take care of her lawn. I am not asking that he never speak to her or he drop her as a customer or that he even stop being friendly with her...just back off. He was beyond furious that I tempted to "control" his life when he has done nothing wrong. It was an awful night. The next day he work up and acted as if nothing ever happened. I feel like I am walking on eggshells. Just waiting until I ask the question that is "too much." He told me I don't ask enough questions...I am just lost.  Understand this is not my fault and in a healthy marriage, he would say "I am not doing anything wrong but my wife is the most important person to me, so I will step back from this friendship so she can see that." I understand he isn't doing that.
Now to the real question...
We are actively TTC. He was a little freaked out when we decided to start trying, but he said he was on board. We hit some bumps here and there...I can be a bit single minded when I have a "project." At first I was being too much and he felt a lot of pressure to "perform." I have backed off a lot and we are having trouble. Low progesterone and estrogen. I saw the reproductive endocrinologist yesterday. When I made the appt, they requested my spouse come,  but said it wasn't mandatory. Of course he said he couldn't make it. They sent me a cup home for a semen analysis and said we needed to call to schedule it. I asked him at noon about a time and he said he couldn't get to,his schedule then, then he couldn't think about it at 8 pm either. He didn't want to hear about my appointment and was pretty noncommittal when I told him I would need him to drive me to and from the hsg test bc I would be doped up on Valium. He says he wants this pregnancy to happen...his actions just are not showing. 
I got to thinking yesterday, but all of this started when we decided to TTC. Do you think he is just freaked out about the possibility of a baby and acting out? The behavior escalated when the referral for the RE was made. It almost seems like he was a little freaked out, but it was ok bc we were having trouble, and now that things have progressed to us really getting a handle on what the problem is, he is panicking? I feel like this chick is almost like a last cling onto a life of being carefree and independent.
I know he does this. He did it before our wedding, but chilled out afterwards. It's just recalling screwing with my self esteem and my stability. If you think this is the case, what can I do to help him see he is ready and we will be ok?
Please, don't tell me how this is emotional abuse, how I need to leave him, how he is having an emotional affair, that we have no business having a baby with these types of issues or anything else negative. I know these things. I am not going to leave him, I choose to stay in this because I know he doesn't want to treat me this way. I am not going to give an ultimatum and I am not going to contact this woman. Just help me figure out how to reassure my husband- who I love dearly and respect greatly- that we are as ready for this as anyone ever is and that we will be excellent parents together. 
Thanks! ❤