Old co worker wants to have lunch with me

Ok so I am a say home mom now and my husband is a truck driver. He’s gone on week days. I love my husband even though we fight sometimes. But I mean what marriage doesn’t have flaws? Been married for 4 years and two kids. Anyways my old job had a lot of males. I was the only female and it was fine because no one ever made me feel uncomfortable. My husband knew I had guy friends and he knew my girl friend left to be a stay at home mom too. I had this guy friend and he is straight up only a friend. Like we always went to lunch together and would find each other at work to talk about our weekend and talk shit about the people we worked with like usual. So we only talked at work and kept it at work. Never any texting or phone calls, nothing. Before I left the job we were like hey we should still keep in touch and have lunch. It’s something people say but never really follow up on that. But idk I changed my mind now. I don’t want to see him ever really, only like if I run into him at a store. I feel weird because he was only my friend at work. And now that I don’t work with him anymore it feels different and I always felt like he kinda liked me but he never made any moves on me. We were just really close friends and we listened to each other’s problems. I don’t want to see him anymore because I don’t want it to open doors to a different kinda friendship. I have a good trust relationship with my husband and I just don’t want to give off the wrong impression. Ugh it’s hard to explain. He asked if we could have lunch this week and I said no I’m busy maybe next week. ( I wasn’t busy) I said whenever. So now as of Monday I’m going to have lunch with him. I don’t wanna go! I’m NOT going. Idk what to have said. Also he sent me a text. We never texted each other. Should I ignore Him? Block him? Keep making excuses? I’m probably over thinking it. Help?

Also I know it’s just lunch but he has had a history of cheating on his wife with other women. He told me he physically cheated on his wife one time and I felt horrible and I was mad at him. I was like your my best friend but I really wish you didn’t do that!! 😧 I asked if the wife knew and he said he told her but they weren’t doing so well after. They are still together.