Disagreement on doctors appointments

So my husband and I have been married for 4 years and trying to get pregnant 2 of those years we recently took the next step and got a semen analysis done on him which he was not happy to do and it came back abnormal which made it even worse. The problem lies in that he does not want me to be in the room when he discusses the situation with the urologist. He says this is his body part therefore it affects only him not me and refuses to have me involved in it. I feel that we should do all of this together since it is for our future and our family and this started a huge fight.

Can anyone give me advice on the situation and what I should do?