Sexual Abuse?

My dad brought us to a babysitter when we were young and I'm starting to remember things more and more. I remember this man, the babysitters son, coming into the bathroom and touching my breast and also taking pictures of me when getting dressed. I vaguely remember that when I was young but it's coming clearer. I also remember this same man taking his pants off in the basement, but that's all I remember. I have been really depressed lately and having terrible nightmares of this. 
Why is it al the older you get, the more you remember things? I don't want to get any older cause these memories are haunting me. I was Maybe 4 when this happened. I am now 27.