Should I be worried...

My SO and I have been together for nearly 4 years, anytime I bring up the idea of engagement or of a baby (a few years away) he clams up and even gets upset saying "we're not ready!"  Should I be worried that he won't even talk about that part of our future?