Condoms and the pill

I'm on the pill but my boyfriend and I still use condoms. He hates them though and would rather not. How safe would I be though being just on the pill? My paranoia sets in and I think the worst, but others do it all the time and they're fine? 
Just talk to me haha! Advice