It is me... or is it him?

Im curious. Me and my bf have gotten pregnant three times. And all three times happened with only having sex once, and not <a href="https://play.google.com/store/apps/details?id=com.glow.android">tracking ovulation</a> or anything. Each time we randomly decided not to pull out. The first time we got pregnant and thought wow we are lucky. Had a sweet little baby. The second time, same thing. Sex once, pregnant. This one ended in a miscarriage. Right after the miscairrage, we decided to start trying again right away without waiting for a full cycle. We had sex twice this time since we have no clue when i would ovulate and we werent using strip or temps or anything. And we got pregnant again.

It has made us wonder... which one if us is really fertile? Or is it just a combination of us both.