Trouble in work place

So, I screwed up once at my new job that I've been working at for only a month now. It wasn't anything major. My boss said I told him I would work yesterday, but I don't ever remember telling him I would. So, when he came in today, he said "you were supposed to be here yesterday. I talked to you about it and you said you'd work." & all I said to that was "Oh, I don't remember the conversation. Or I of course, would have been here." . I literally DO NOT know what he's talking about. I never talked to him about working yesterday. And he was like "Well, I'm just saying, I don't make mistakes like that, so I would know." Ok...?

It's a $10p/h job. Of course I'd be here if I KNEW, to make that money. I feel like he's accusing me of agreeing to it, WHEN I NEVER TALKED TO HIM ABOUT IT. I'm a little upset, like kinda frustrated because I don't want them to think I'm not reliable.

What would you say to your boss(general manager), if this happened to you? I'm obviously going to move on, but it's just stuck in my head right now.