Argument with my husband over nephew

My husband and I have been together for 14 years and married for 6 of those years. We recently got into an argument because of our 4 year old nephew/Godson (my sister’s son). My sister is a single mother who works as a server. She took a new job recently at a restaurant and has asked me to help watch my nephew for about an 1.5 hour in the afternoon until my mom can come scoop him up after she gets off work. I have the flexibility to watch him because I work from home and he’s a good kid that literally just plays with his toys or things he finds in my daughter’s playroom. He’s my nephew, I love him like my own child and there was no question about it since it’s not a disruption in my day. I am home alone during the day as my husband goes to work and my daughter goes to my in laws during the day so I can work. For clarification, she is 2 and does not let me work which is why I need my in-law’s help with her.

My husband got upset over the temporary arrangement until my sister finds a school for my nephew that she can afford. I told him I wasn’t sure why he was upset. He proceeded to tell me that when he gets home from work he just wants to relax and my argument is that he will likely not see our nephew and if he does, maybe 15 min (if that, depending what time he’s off work). I told him it wasn’t a bother to me, and he said it’s a bother to him. He then told me that if my sister continues to bring my nephew he’s gonna have a talk with her and my mom.

If the tables were turned around, my mom and sister would help us out no question asked. This situation is really making me question our marriage and on top of ongoing and recent fights between us has been the cherry on top. I don’t like the person he has become. The initial fight over this happened two weeks ago and things have been weird between us. We only communicated about our daughter’s needs. Then he tried acting like nothing was ever said. He was trying to be intimate and to be honest, I do not feel attracted to him at all (after this fight and how he’s been acting recently) and declined his attempts.

Well when my sister asked for my help this week, I told him this evening I would be helping her and the argument came up again. Now he’s upset with me. Am in the wrong here?? Please help!