Advise please!!

My husband and I live in a house that he bought a couple years before he and I met. It's small, and would need an addition before we decided to have children. He loves it so much, which is why I feel so bad for wanting to sell and buy a new home! Am I wrong for feeling this way? I just don't feel an attachment to the house in the same way he does. I want to find a home together and make it "ours". We have discussed the situation many times and he is open to buying a new house, I just can't help but feel guilty because I know how he feels about the one we live in. Thoughts?