When is he right time to set a date ?

I got engaged recently and of course people ask if we have a date set. My fiancé and I agreed that we are going to plan it our wedding together. When people ask us he says that it's too soon to think about it and I agree for now But when is the right time to talk about it with my fiancé and what do I say ? I don't want to pressure him

FYI we've been together for over 4 years