Is he right to ask me not to see him?

So long story short my male best friend is also an ex of mine, my boyfriend doesn’t like this, we don’t see each other as exs despite this my boyfriend does not like me seeing him, my family think he’s being controlling so I need advice..