He accepted a phone number

Kimberly
My boyfriend is a musician and 3 nights ago he played a show. Some girl playfully put her number in his tip jar with a sugar packet. At the end of the show he was counting the tips and found her number and threw it on the floor. I didn't think anything of if because he tossed it along with the sugar packet. I stepped away for maybe a minute and when I came back the number was no longer on the floor. I asked him where it went and he looked around and picked up the sugar packet and said "found it". He was pretty drunk and he stood up and leaned on me and said I had to help him get out of the bar without looking like a drunk asshole. I put my arms around him and my instinct said check his pockets and sure enough, there was her number. I asked him why he kept it and he acted like he didn't understand what I was asking so I repeated myself and he did it again. I'm not going to let being drunk be an excuse, he took the number and I know he wasn't going to throw it away because there was a trash can right next to him. He took it and he was going to use it. In my mind he had intended to cheat. I mean, really, what else was he planning on doing with it? Challenging her to words with friends... Right...
Is it far fetched or wrong for me to feel this way? I can't gauge if I'm being over dramatic, and we haven't really talked since that night, but he's acting as though nothing happened.
Side note, were in our twenties, we live together and have sex maybe once every two weeks and he turns down nearly every sexual advance I make. (He even turns down blow jobs, I don't get why I'm not ugly by any means and we used to fuck all the time) so if someone else offered him the option, how am I supposed to believe he wouldn't be interested. Really any advice would be awesome. It's hard to talk to him because he gets sarcastic and cold, like my mother, and I just clam up.