did I ask him out too early?

I have been talking to this guy and we have been getting along so nicely. It's only been like 4 days... but all I said was does he ever want to hangout with me. He always replies but I'm worried I asked too early!!! Way too early! I just like him and want to get to know him more in person..... was this a mistake