Sore boobs

I was on the birth control shot until recently when I switched to the pill (months ago). I hadn't had my period in 3 years but it's has started since I started the pill.
 My boobs have been hurting for months now, but only when I'm NOT on my period. Sometimes they hurt so much it's unbearable to wear a bra... Should I be worried, or do you think it's just my body adjusting?