Question - Am I right or wrong?

Long story short, my grandad is having surgery Monday and right now I have his car. My grandparents don't let anyone else drive their car except for me really. Ok so when I got the car it had half a tank of gas. My n my BD went to eat and see a movie. Saturday he had choir rehearsal and he put $20 in the gas. I didn't use the car at all Saturday. So this morning my BD wakes up and tells me I don't have to go to church and it made me upset (we go every Sunday and we got into a fight last night which is why I assume he told me not to go) At first he was telling me he will give my the car keys so I can leave. Well half way they the conversation he turns around and says I'm not giving u the car keys, he put the gas in it and he won't give me the keys back until I pay him back the gas money (I also don't work, I'm stay at home with our daughter). He then proceeds to get in their car and leave me at home. My mom bday is today and my 85 yr old grandad is having open heart surgery tomorrow and I don't know what to do. My BD ran the gas out in he car and replaced it but won't give me the car keys til I pay him back!?! He says the gas was for the weekend so he could get to church and the only reason we're having a problem is because he told me not to go, if we were going together of course we would use the car! I just don't know if I'm crazy or not? He ran the gas out and replaced it but I'm supposed to pay him with money I don't have? Then he says he will give them to me when he gets home so he can "show me" and lock the door when I leave. Just frustrated and not sure what I did that was so bad I would NEVER take someone else's grandma mama uncle cousin keys and refuse to give them back wtf!!!!