Bf keeping me from seeing my mom?

I've had plans to go see my mom since last Saturday. I woke up late this morning (10am) and I have to be ready at 12pm so I'm kinda rushing. My bf asked me to go outside with him to smoke so I did and while we're out there he asks "so are you coming back tonight?" Keep in mind just last night we went over all of this and I told him maybe because I'm not sure what we're doing. Then he goes "well you better figure it out" and at this point that REALLY triggered me because I'm my own person and he doesn't control when I see my family nor do I ever have to ask for his permission.. I told him exactly that and then he's like "you don't even have to go.. you were just complaining last night about me taking a job" I only complained because he wants to take this Construction job where he leaves ALL WEEK and is only back on weekends. Even tho he has MANY other options he was suppose to take locally he's wanting to take this one which is lower paying so it doesn't make sense to me. We currently live with his parents and I left my home to come live with him so if hes gone all the time I'm not gonna stay here. I don't even have my own vehicle yet.

Am I wrong for feeling this way?