Confused

I've been with my boyfriend for 4 years now and for the last year we have been talking about starting a family.. we have been off birth control for a year and I finally decided to make an appointment to make sure everything was okay to have a baby and start this adventure. Once my partner found out he told me to cancel the appointment he doesnt think he wants a family any time soon...am I wrong to feel different about our relationship now ? To feel like were just going down a dead end ...if my plans and his plans dont line up then wont it just end up driving us apart ?? Advice please