Am I wrong

I feel like I need to give a few details in order to ask this question. I am married for 2 years now my husband has two children. He has a very strong bond with his daughter over the years we've gone to pick her up several times together. Recently he has been able to spend more time with his son and the other day we were riding in the car we stopped at my mother in law house whom lives right down the street from us and we were standing outside talking to his mother and his niece when he tells me that he's going to pick up his son for a few hours so that he can spend some time with him. I said ok he walked in the house and did something for his mother during that time I set back in our car in the driver seat I don't drive normally I am a passenger rider because I don't like to so when he came back outside with his nephew he asked me to go into the house and sit with his mother while he goes to pick up his son and it upset me. Because I had never had to sit in the house to pick up his daughter. so I informed him that this would never happen again from now on if I'm in the car we can go together it's different if I'm already in the house he did not understand that then later he tells me that it was to keep confrontation down because he do not want to cause a rise in his son mother. I explained to him that our marriage is something that she needs to accept and that causing trouble in our home to make her comfortable is not an option he does not understand he feels that I'm wrong that I should not put our happiness in someone's face am I wrong for feeling like this. And just to keep it fare because I really need to know am I overreacting I should also say that in no way do I feel that he would cheat on me with either one of the women is not about cheating it's about respect. His son lives around the corner. It literally to about 2 minutes. As I said it's not about trust. I felt disrespected. Would anyone else feel this way. Am I wrong.