Late Periods

I've been on the pill for about 3 years, routinely I would get my periods on Thursdays, and they would last just about 5 days (until I started taking my active pills again). However, my periods have been coming progressively later and later, most months requiring me to have sex to get them going. I feel like this is so abnormal, but my doctor told me not to be concerned until I miss my period 4 months in a row..I feel like o should get a second opinion? What do you ladies think? Sorry for the long post.