Boss probs

So I just started this new job about a month ago. It's been going great. However I just found out the manager had an affair with the last girl in my position and it's making me uncomfortable. I worry he'll want it from me too. I feel like I want to leave the job now. Or should I just watch my back and wait and see what happens? I'm just really freaked out now. I'm not into that kind of thing.