Am I wrong?

Me and my bf live together. I’m 21 weeks pregnant and we don’t really go out and do much. My brother texted me and asked me if I can watch my 2 year old nephew tomorrow while he’s at work. I said yeah I don’t mind. I told my bf that my nephew is gonna come over tomorrow because I’m babysitting him. My bf got all mad and said I should’ve talked to him about it first. I got a little defensive because I felt like I didn’t do anything wrong? It’s not like I had plans to do anything else but sit at home. He said it’s his apartment too and that I should’ve asked him first. Am I wrong for not asking him first?