Only having 3 kids?

My husband and I are 23, planning baby #3 in another year or so after we have built our home. We keep going back to the conversation because one of my siblings and one of his had 4 kids and we just love seeing their big family. At some point we even talked about having 5/6 kids! We were pretty convinced that after baby #3 I would burn and tie my tubes lol but lately we’ve been talking and we feel like we might regret not growing our family more. Has anyone only had 3 and wished they had more? I know everyone is different I just want to hear other peoples stories