I no longer want 4 children. Is it fair for my husband to be upset?

We have 1 child. Shes 16 months. Being a mom is lovely. One child is hard work but manageable. We are currently TTC for baby no.2 and I’m excited at the possibility of bringing another life into this world and having a sibling for our daughter but after this I am not on board with our original plan/desire to have 4 kids.

My husband thinks I’m being selfish by backing out of the plans but these plans were made before I had my first child.

We have been together since we were 22. We are 30 now.