Am I over-reacting?

Trigger Warning - child abuse

Back story - I met my current partner when my girls were 3 & 5, we used to catch up with his sister, her partner and daughter quiet frequently - her partner was always with the kids, would take them into the ocean, play hide and seek in the dark, take them to the park on his own, I thought he was a nice guy and just loved playing with the kids but when my oldest was 7 I found her in a room with the brother in law she had a dress on and her knickers down at her ankles. I was in absolute shock and couldn’t speak - I questioned her if he had touched her or if they had any secrets I googled lots of things and she always said no nothing ever happened and then I thought about all the times he had her in the water and he would ignore his own child and give more attention to mine and all the times he would take the kids to the park on his own it made me sick but I was fairly confident maybe we found out what he was like before anything did happen so we decided we would have nothing to do with them anymore and we haven’t for about 4 years..

Until yesterday :( in those last four years we haven’t seen them my partner and I have had two boys of our own, my partner went to his mothers house and the sister and brother in law were there, my partner said he shook his hand and said hi to them, I was pretty angry about this and my partner thinks I’m over-reacting and his opinion on it is “but nothing happened” and what was he supposed to do just ignore them? It makes me sick and angry that he could shake someone’s hand after that and if it was his own two boys it happened to would he think differently about it? I don’t think I’m over-reacting I think any normal parent would feel the same? I would really appreciate anyone’s thoughts on this, thank you