Missed period

So back in July my boyfriend and I broke up so durning that I finished a pack and automatically started a new one. I didn’t take the brown pills. I stopped taking my second pack because i realized i didn’t take the brown pills because i was stressed out. So after i stopped my second pack On the 23rd I started my period which lasted 5 days. Come August I missed a couple pills when I started my second pack up and it’s now September and I still haven’t had a period. I am 23 days late and i be taken 2 pregnancy test and both said negative. My doctor said my body is confused because I’ve been missing my pill but also I started and stopped my birth control when I should have continued to take it. Should I be worried?