Trying to conceive for 20 months and IVF

Hi everyone, my husband and I have been trying for 20 months now and have been advised that the best option would be to go down the route of IVF as hubby's swimmers aren't playing ball.  I am unsure on whether to hold off for a few more months before  being referred to the specialist and see if it happens or just go for it now. Has anyone, who has been told it's unlikely to happen fallen pregnant before going for IVF?? Needing some advice and a little positivity right now. Thanks xxxx