Wedding guest list???

So my man and I are getting married in April. I told him since we first met my ideal wedding would be a small intimate one. I rather have quality rather than quantity. How do you choose exactly whom to invite and whom not to invite? I don’t want to have people mad at because I didn’t invite them. But just because I know you doesn’t mean we’re close. I have very distant cousins, my brother in laws family (who I’ve known for years but aren’t really my friends or family), 2nd cousins, etc. Also my brother and I haven’t spoke in 2 years and we live in the same house. He gave me a black eye for trying to defend his gf at the time so I don’t see myself being cool with him(he never apologized). Would it be wrong to not invite him to my wedding? I want a Kim Kardashian vibe wedding where it’s very small, no plus 1s whom I don’t know.