Ex at the wedding??

My ex boyfriend before my fiancé happened to be one of my fiancé's best friends. There was no affair or any kind of drama - ex broke up with me and my fiancé and I started dating about 6 months later. 
Fast forward nearly 4 years later and my fiancé and I are planning a weeklong destination wedding with a small group of close friends that we rarely ever see because they are across the world. We don't think it's a good idea to invite my ex because it is essentially a weeklong vacation ending in a big celebration of our relationship. I think he may be expecting an invitation, however. 
How do we tell him that he's not invited? I don't want to sound rude but we don't want him there