Husband goes to lunch with a co worker of the opposite sex. Should I be worried?

He says they are just friends and I never thought anything of it when he would say him and a buddy and a few girls went to lunch, but when it's just him and one woman, it bothers me. I wouldn't go to lunch with a guy by myself out of respect for him.