No weight gain, but pants don't fit :-(

Sarah
I fluctuate between the same 5 lbs as I did before I was prego, however my pants do not fit. My husband says I have lost weight from my love handles though. I Am heavy set to begin with, so I feel guilty for not firing in my clothes. Anyone struggling with this as well?