TTC with one tube & male factor infertility

In October 2018 I became pregnant for the first time . Unfortunately I miscarried around 8 weeks(blighted ovum). In early April 2019 I found out I had an ectopic. My tube was severed and no longer functions.

Since then, my doctor recommended I go to fertility doctor. All tests came out favorable. Doctor said I could do

IUI for a boost or still conceive naturally.

In the meantime, my husband had a sperm analysis which indicated he is low in motility, morphology, and concentration. Based on results, they are saying he has male factor infertility. They now say we should go straight to IVF and IUI will likely be unsuccessful

.

Isn’t that jumping the gun given I was pregnant twice? His sperm can’t be that bad?