My SO keeps using me as a excuse to his friends

The other day my SO friend called and asked him to come over to chill. My SO tells him he can’t because Stephanie (that’s me) wants me to hang out with her tonight. His friend then invited both of us over and he said he will talk to me.

(I could hear the whole conversation because he always talks on speaker phone And I never said I want to hang out with him tonight or to stay home).

I asked him if he want to go over there and he said no he just wants to chill at home. A little bit later his friend calls back and asks if we are coming over and he says not tonight Stephanie just wants to chill at home.

He literally uses me as a excuse every time he doesn’t want to go out with his friends. This pisses me off because I never tell him no he can’t. But to all his friends it probably looks like I don’t let him go out with them.