Late period...

So my SO and I are frequently having sex, without a condom. I was on birth control until my insurance dropped and the price went up and I could no longer afford it. My period is about 20 days late as of right now... should I start freaking out about pregnancy.? I've been pregnant before but I lost my baby about 2 months into the pregnancy so I've convinced myself my body won't allow me to have babies... my boyfriend noticed I haven't had as he calls it "shark week" in a while and asked me if I was pregnant last night... How much longer should I wait to test or should I test now??