I don't know how to take it

An

So my SO has two kids with his ex. When he has them it's all fine and I enjoy being with them, but when he has to take them back to their mother he always asks me if I want to come with him. The only problem is that every time he jokes that I have to hide if I go with him. She knows about me so I don't know what to make out of ti. Ideas?