Tmi but I'm so scared

I did it.. I gave in. I'm 3 weeks 4 days postpartum and my husband and I were messing around and as things started getting hot and heavy we started to have sex (couple in and outs sorry tmi) then we actually both got scared and stopped. It didn't hurt, but now I'm just completely freaked out about an infection and also getting pregnant again. I know, I'm so bad for not using protection.. Anyone else go through this? Should I be worried? I'm literally losing sleep over this. :/ I'm not sure if it matters, but I am exclusively pumping and giving baby breast milk. I heard there's less chance of pregnancy if you're breast feeding? Idk.