IVF baby number 2?

So my husband and I have an 8 year old boy that we conceived naturally and a 13 week old baby boy that we conceived after almost 3 years of trying, procedures and a round of IVF.  We have been told that our 8 year old was a miracle and that going forward we would only be able to conceive through IVF.  So here is the question, should we go for baby number 3 (IVF baby #2)?  Is it too soon to be thinking about this?  I guess I am just trying to prepare myself for more IVF nightmares.  Any advice or thoughts?  What has anyone else done?