skirts to wear or not...?

So we used to go to a church a few years ago, where the pastor was strict on women wearing skirts. Later we found he was commiting adultery with 2 women. So I really don't take into consideration what he really said. My husband when he is in his religious mode he becomes upset if I don' wear skirts to bible study, I don't see anything wrong with wearing pants or jeans to bible study, now church I understand. Is he being a little over the edge? Most of time I go with want he wants just avoid confrontation. What do y'all ladies think?