baby daddy literally went off on me

So tonight, i told my baby daddy i was going to sit in my car and listen to music (we aren’t together our son was sleep) i have him staying with me in my guest bedroom temporarily.. so when i stepped out side i called my aunt to go see her because she JUST HAD A BABY and was alone. i went to go see her and didn’t tell my baby daddy, he texted me saying “i hope your having fun” and he demanded that i tell him where i go, when he’s staying in MY APARTMENT WHERE I PAY FOR EVERYTHING am i wrong for not telling him?