When to tell people you are expecting

When did everyone start telling people they were expecting? 
This is my second pregnancy and with our first we told people almost right away. 
However, since we now have a 5 yr old little boy who has been praying for a sibling for over 2 years now, I worry about telling too soon. I'm only 4 wks + 1 and my hubby really wants to start telling people. I hate to be a worry wart but I don't want something to happen and then we have to tell our son different.