TTC advice with one fallopian tube

In October 2018 I became pregnant for the first time . Unfortunately I miscarried around 8 weeks(blighted ovum). In early April 2019 I found out I had an ectopic. My tube was severed and no longer functions.

Since then, my doctor recommended I go to fertility doctor. All tests came out favorable. Doctor said I could do IUI for a boost or still conceive naturally. In the meantime, my husband had a sperm analysis which indicates he is low in motility, morphology, and concentration. based on results, they are saying he has male infertility. They now say we should go straight to IVF.

Isn’t that jumping the gun given I was pregnant twice?