Hubby being haunted by my grandpa, I think

Weird story. My Grandpa passed away in the early 90s (he was born in 1901). He was an avid rock collector, I mean, he'd go mining for different rocks and had a whole room displaying his finds. The only thing I inherited from my grandpa were some rocks my grandma let me pick out. That was okay, I loved those rocks. Fast forward to the other day, I was cleaning out my sex toy box, which I hadn't done since we moved. I found some of my grandpa's rocks at the bottom. WTF? Maybe they ended up there when hubby was packing in haste? So, I cleaned up the rocks and put them in my rock display, which includes other rocks of grandpa's my son added to. The next day, my husband says he saw a tall man in our closet, dressed like he's from the 40s or 50s... He's seen him just about every day since, turning my skeptic hubby into a believer. I didn't put two and two together until I was clean the rock display this evening. I think my grandpa is unhappy my hubby put his rocks with our sex toys!!!

How do you deal with that??? Most embarrassing haunting EVER. I remember apologizing out loud to my grandpa when I found his rocks in such an inappropriate place. But maybe my hubby needs to, too.