My director at work told my manager about my fertility issues

We have no HR person at work our Director Of Operations wears many hats. I told her that my husband and I was going to see a fertility doctor so I may need to adjust my work schedule..take longer lunch break however my time will be made up..so we got a call back from the doctor she allowed me to have a early lunch break when I got back I went in her office closed the door and cried my eyes out. I did not mention to her not to mention this to anyone else because I was crying and as a women I would've never thought she'll tell. My manager loves to gossip hence why I opt to the the Director and not her. When i request to have an extended lunch I say I have an errand to run and I email my manager and cc'd the Director again I never say the reason being. I recently spoke w/ a friend that no longer works for the company she mentioned that my manager has told her that I'm having fertility issues and she wanted to let me know that it was pure gossip. I'm so furious that a Director will discuss my personal business. All I think of is how many at work knows about this. I want to confront them however I'm afraid I may burst.. she literally betrayed me. Ladies how should I handle this.