Maternity pants trouble

So a couple weeks ago when even my stretchy pants became too tight I caved and bought maternity pants. I'm normally a 8-10 so I got a pair of medium maternity pants. At first they fit well, but after a couple hours they start to slip. I have to constantly tug at them, so I exchanged for a small thinking this would solve my problem. Even the small sags! Is this normal? I'm currently 17 weeks and 4 days, but already have quite a bump.

Folding down the elastic band doesn't help either.