am I wrong?

So I've been on the pill for about 4 months now. I've never missed a day but have taken it a couple hours late a few times. Taking it late probably won't make me ovulate, but my boyfriend is extremely paranoid if I even take it a half hour late and insists on wearing a comdom and it's getting annoying. I'm 21 he's 27. He had gotten his ex gf pregnant while they were still in college about 2 years ago. We've been together a year 4 months (she had an abortion) so I guess that's where it stems from. We do want kids together but in a few years. Is he right to be so paranoid? Could I get pregnant from taking it late? He cums inside me 95% of the time