Seriously don’t know what to do

So me and hubby have been TTC since November (with no luck) when we got the ok from dr after my hysteroscopy to remove tissue build up in uterus that they discovered after I lost my baby at 15 week. well last night he got a call from his aunt who is again in jail she had a baby less than a year ago and the baby was taken away cuz she is a drug addict and now she’s asking us to adopt her baby cuz no one else in family will. We have nothing to do with this aunt for the reason of she is a drug addict and I know that if we were to adopt the baby she would try and be around all the time. I am so frustrated and upset cuz if we do this (which we honestly are considering but talking about it) his aunt will try to be around all the time even tho we told her that if we did she would not be allowed around all the time just holidays and family get together sad it’s hard to avoid that time with family. I just don’t know what to do. And I am worried that this will halt me and hubby trying for another one for us and I know that sounds selfish but I have a 6 year old with my hubby and we have lost 4 babies to MCs and the one at 15 weeks was just over a year ago