Is it stupid to get my hopes up

I just recently got married, I have 2 kids and my husband has 2 kids. I really really want another baby so we can have at least 1 together, my husband however isn't 100% into the whole idea of having 5 kids. Do I keep hoping maybe he will change his mind or should I just give up and disregard how badly I want another one