When I was 6...

When I was 6, my older brother had a friend that would run his fingers up my shorts and rub my thighs. Sometimes he went near my private area and he'd always smiled at me while doing it. He never touched me in my private area, but it caused me so much anxiety because he was over at our house every week and I hated seeing him. Do I have reason to still be very upset by this even know he didn't actually molest me?