Am I wrong in feeling this?

So I mentioned it would be nice for him to take me and our littlest man out for lunch. And so today rolls around and I asked him if he was and he said sure and we went out and ate and it seemed really nice and little one enjoyed being out.

On the ride home I was texting a family member and said that the three of us went out and had lunch because we wanted too and he was like "all three of us? What do you mean all three of us" and I said yea we did and he was like "no you two wanted to go not me"

It kinda hurt my feelings that he didn't want to do something with us. Is that a wrong feeling?