Missing/taking pill late

To be very honest I'm really bad at taking my pill I probably miss taking it about 2 days a week and I usually take it late the other day (I work late and never bring them with me) me and my boyfriend don't use any protection either. I've been with him for almost 2 years and I've been on the pill for about 4 years. I still get my period the same time every month unless I really miss to many days and then I will get it sooner. Me and my boyfriend have never had any pregnancy scares or anything and I've never even thought of being pregnant. Everyone always makes it a big deal that if you take it late or miss it basically you're fucked and will get pregnant, well since it hasn't happened to me I'm wondering if there could be a problem with me possibly not being able to have a child or is it just luck? Can any relate or might have some more knowledge on missing pills?