Don't want kids anymore...

La
My husband and I are both 28...will be 29 this year. We have been married three years, together 4.5 years altogether. When we first met and got married, I was always if-y about kids but agreed to give him at least one. 
This summer we have spent a considerable amount of time on vacation with my 1.5 and 3.5 year old nieces and his 8 year old niece. These kiddos have been really rambunctious, bratty, etc. made us enjoy leaving and getting away from the choas. We both felt that way.
So tonight, after dealing with his rambunctious 8-year-old niece, I said I do not want kids. Period. He became a little ill, said I can't base that on other kids. Am I wrong for changing my mind, or should I follow my heart, even if he isn't necesRily on the same page??