I removed him from my instagram

So I used to see this guy but things didn't work out. I barely talk to him anymore, we're on good terms but we haven't been close.

I have a private instagram that only my close friends follow. Since I'm not close to him anymore, I decided to remove him from my followers (I still follow him).

He messaged me asking why I removed him, and I said I frequently delete my followers because I like to keep my life private. But he got quite offended about it saying we're still friends. Would you be offended if you were him? Am I wrong?

I have some friends who asked me why I remove them as well. The thing is just because you don't follow me on instagram doesn't mean I don't see you as a friend?