Endo & Clomid

My doctor said the next step for me would to put me on Clomid. But from what I've heard those two don't go together. I go in for an appointment in a couple weeks. But I'm not sure I feel comfortable taking something if it will only make my endo worse. I got a second opinion from a family friend who is a fertility doctor and doesn't seem concerned. What have you guys experienced? Should I suggest Femara instead? What can I ask my doctor to ease my mind? I know I won't take it if I'm not comfortable but I just don't know enough about it either.