Saggy boobs

I've never been a big fan of my boobs, but they were relatively perky. Since being pregnant, they have grown a couple cup sizes and have started to sag. Is that normal? I thought the sagging didnt start til after birth. Why now?? I hate it. My husband doesn't seem to mind but I'm so self conscious.