left baby with father for about 2 hrs

So my baby is six wks old and I had to run to my mothers which is 45 min away. My BF said to leave the baby with him and his son who is 11, so I did. Baby was just fed and just woke up from a nap and diaper changed. My mothers house is about 30-40 away. I literally went there and came straight home. I got back and he *****ed about me being gone for almost 2 hrs and how I ruined the wkend with his son (11) who we get every wkend by he had to give his attention to the baby for the 1 1/2hrs I was gone. Mind you I am home everyday with my baby and don't go anywhere. I feel like he is being unreasonable it was about 2 hrs there is breast milk pumped in the fridge and everything he needed for the baby. I get he only gets to see his 11 yr old on the wkends but you can't deal with the 2 of them for 2 hrs alone am I wrong. And yes this is something I couldn't wait to do I waited two month for me to be able to do this.