Maternity pants too soon?

This is my second pregnancy and I'm only 5 weeks but all my elastic bands are so tight. When did you start wearing maternity pants and is it crazy if I start soon! Not even sure mine will stay up because I definitely don't have a belly it's just so uncomfortable.