Am I wrong for being suspicious?

Long story short.

I was out in Oslo this weekend with my girls and met some friends. When I got home on Sunday I saw a pink sweater laying on the bed. I thought nothing off it (because me and my boyfriend also share house with his mom ,just for a while), so I thought it was hers.

So I just placed it in her closet. Next day she says she put the sweater in my closet.

So now I know it doesn't belong any of us.

I asked him and he said he didn't know.

I am super suspicious because he had the house all for himself this weekend and could easily fool me.

I dont want to push asking him because he gets super angry at me. But I have no other conclusion that he had a girl over.

I also double checked with my friends and asked if it could be any of my mother in laws friends, but nope...