Hangout with an old friend of opposite sex?

So, I have a friend from high school (J) who fairly recently moved back to town. We were pretty close back then. My husband (S) met him back when we were dating and they hit it off well. J is in a Marine and has been on a couple deployments but we all kept in touch when he was gone and hung out when he was in town. Now he’s back in town for good and we rarely see him. He called me today asking if we could go to lunch. I called my husband to see if we could go, but he was pretty swamped and told me to go. I told J that S couldn’t get away, but had said the kids and I could have lunch with him. Only my mom had me house sitting because she had a carpenter working, so I couldn’t leave.

J went to the office to get money and say hello to S and then got me and the kids lunch and brought it to my moms. We sat outside until it was kiddos’ nap time then sat inside on separate couches and talked. My mom, aunt, and grandmother showed up and we all talked for a bit. My mom called me a few minutes ago and told me that it made the conversation awkward because I had been sitting in the house for a while with a man other than my husband, without my husband.

I can see where it could be suspicious, but he’s been a friend for so long and he’s also one of my husband’s buddies. Nothing EVER even came close to happening between us in all the years we’ve known each other and I would never dream of cheating on my husband. I guess I just don’t get what all the fuss was about with him.