What should I do. KIND OF LONG

About 3 weeks ago my fiancé and his family went to a party, I didn't go because I was working that night when I got out it was raining really bad and I called my so to make sure he made it there ok, I called and called and no answer. So I called his parents, older sister, and little brother and no answer. I saw his sister was on fb so I msged her and it turned out to be her husband that works out of town and was still in his apartment where he works. He ended up calling me through fb and then asked me if he could call my phone which I said yes to because we were having a conversation about problems with our in laws. He called my phone and after awhile he started getting a little flirty, telling me I have a nice body, he said if he had met me before he met his wife he would of dated me, and he also invited me to lunch but he said just us two and my so and sister in law couldn't find out, I didn't like it so I ended up telling him I had to go to bed because I worked the next day, and no so and I don't live together yet. Anyway he hadn't bothered me since, and I haven't seen him thank God! But today he's called me at least 8 times and it's 2 am! He's texting me telling me to answer him but I won't. He has two kids and is married to a girl I see as a really good friend! I see him as a brother, I don't want this to end bad which is why I haven't told my so or his wife because of their kids mainly. I'm getting annoyed and don't want things to be awkward, what can I do??