Should I be worried?

So I stopped taking oral birth control beginning of May this year. My periods went back to normal relatively fast. Last month my period was a couple days late so it took a hpt and it was negative. My period came and only lasted about 2ish days. This month I had brownish discharge for a few days and my period started about on time and only lasted two days. I don't know what is causing it to be so irregular so I am taking a pregnancy test again to be sure. I don't have any other symptoms besides the spotting. I haven't gained/lost a lot of weight, I'm not particularly stressed, and I don't have bad cramps. I plan to go to my doctor about it but I don't want to freaky self out before I go, I have the tendency to assume the worst 😩 Anyone else experience anything similar?