Would I be out of line if I said this?

Sometimes clients prefer directly with me than going to the office. That's fine, however today a client messaged me at 07:30 (yes, 07:30 on a SUNDAY) with an accounts query.

I have only messaged him back now, you know on account of it being a freaking Sunday and not an urgent situation and he complained that I took too long to come back to him and it's unprofessional.

Would it be out of line for me to ask him to kindly refer all queries to the office number going forward? I don't want to give people the privilege of dealing with me directly if they're just going to abuse it.