TTC baby number two and I’m starting to worry😕

When I was TTC for My son (who’s 2 going on 3) I got pregnant right away. (Had him with my ex) me and my husband have been trying for another baby and nothing yet 😩 should I start to worry? How long did it take some people to get Pregnant with baby number 2? Should I give up trying? I’m so stressed out.