TTC - Help?

I am 23 years young and have a 4 year old son. My husband and I conceived him our first time we ever.. ya know, and have been together ever since. When we got pregnant with him it was so fast and we were actually taking precautions to avoid pregnancy but still ended up preg. We recently decided it was time to have another one with no luck so far. We have been TTC for 6 months now with zero luck. I have been off BC for a few years now and we had switched to just condoms because the BC didn't react well to my body. So I know there is no remains of that in my system. Why is it that with the first pregnancy we got preg so fast even when we tried to avoid it and this time are not having any luck? Any tips or tricks? Everyone keeps telling me don't try and it will happen, how on earth are you supposed to not try and expect to get pregnant lol. We aren't going above and beyond like we were the first few months but still not avoiding it. Anyways, sorry for the novel and thank you in advance!