need help.

So a few years ago there was a case started against the man who raped me when I was a child.(they had to re represent him his charges due to some paperwork technicalities). Fast forward to this year they had the court that told him what he's being charged with. The new lawyer has contacted me a few times asking to meet up but I have this sudden fear to testify against this guy. I just want this to go away when I first started this case up I was confident I was ready but it's been years and I just can't I can't do it. I don't know how to tell the lawyer I don't want to pursue I don't want to face this terrible man. I'm happy in this point of my life have two babies 1 year and 2 weeks that need my focus and doing this will send me into a deep depressed downward spiral. I know it's normal to be scared but how do I tell the lawyer I want to drop the case.