Pregancy scare???

okay so I had sex at the end of my fertile window (Nov 9th) and this past week I've had my usual PMS symptoms but not as intense (only slight back cramps, headaches, etc)? I took a pregnancy test last night, and it was a very clear negative. my boyfriend SAYS he's infertile, but he always pulls out before he finishes regardless. November 9th, you guessed it, he didn't. I haven't had sore boobs or anything and my period isn't always consistent, but should I be as worried as I am?? he says he's infertile and I wasn't ovulating, but I'm still extremely worried.