Pregnant again after baby

So lots of people tell me u can get pregnant way easier after u have had a baby. Is this really true? I just had a baby 3 months ago and if I decide to have another I wasn't planning on that for at least another 3-4 yrs but everyone's making me paranoid to have sex now! Lol anyone on here get pregnant fairly quickly after unplanned? And yes ladies I'm well aware of ways to prevent pregnancy but me and my husband have used the pullout method for the past 15 yrs and only got pregnant once (planned) I'm just wondering if u have to be even more cautious then before u had ur first? Thoughts? Experiences?