Are we doing the right thing?

So my husband and I have a 3yrs old boy and even before got pregnant we decided to have only one child, we analyzed our lifestyle and everything and decided this (at least for now, we know we can change our minds in future and no problem) but our family started asked for baby #2 and every time we say our plan was only one our family members and some friends call us selfish or say that our son will grow up spoiled and alone ... anyways I just want to know what you guys think about that, and moms of one child how you guys deal with it.