Ever accuse your SO of being shady & been wrong?

Lindsay • I'm 29, married for 5 years (together for 8). My husband and I are first time parents, having welcomed our little girl Avery on 2/17/17 ❤️, she is is the most fun little girl and we adore her!
This morning I found a women's sweater in our guest bedroom that I didn't recognize. My husband has been home sick most of the week so immediately it triggered alarm bells like "oh my god, has there been another woman here while I was at work?!". Then I remembered that my (15 year old) sister slept over last weekend and it could be hers...I texted her to confirm and sure enough, it is hers. I didn't go crazy or anything, but before I remembered about my sister sleeping over I asked him (in a somewhat accusatory tone) who it belonged to. He was annoyed but answered calmly that he didn't know. I feel bad because my husband has never cheated or even anything close; my insecurities come from an ex who I walked in on with another woman. I know it's not fair to jump to conclusions about my husband just because 7 years ago someone else did me wrong...have any of you ladies ever unfairly accused your partner and felt bad because you were wrong?