Do you let handy-men in if you're home alone?

Katie
My husband and I just bought our first house, and I have days free (I work afternoons and nights mostly), so I have been handling repairs/deliveries. 
Today I had the locks changed, and for whatever reason I just had the weirdest feeling about the man working on the house. He made me uncomfortable, so  I trusted my gut and stayed outside in the open while he was working. He stayed in my driveway after I paid him, and I was finally forced to leave after 10 minutes to go to work, so I don't know when he actually left. It just didn't feel right. 
I was wondering if anyone else had experience with this/advice on how to handle future situations/if you'd be comfortable being home alone with a complete stranger?