it's my car

C
So my boyfriend and I have a baby together and he has two kids we have with half the time. He has an old VW that has been costing a lot in the past couple years to maintain and right now it's acting funny. It works, but it's not that reliable. My car is almost new, a 2013 and works great, but I make biweekly payments on it that I pay for. It's my car. He has his car and I have mine. Before I got my car, he did not want to teach me how to drive his (it's a manual), so I never drove it. He was all snobby about my car being that it is automatic but has recently started driving it more and more and now that his car is acting funny, he had been borrowing mine quite a bit. I don't have a problem with that oer se, but this morning he had to drive his kids to their mother's place and he told me he would be taking some personal time before coming home. I don't necessarily have a problem with this except that I feel like he's getting a little too comfortable taking it for longer and longer periods of time. When he has it, I don't lol. I'm stuck home with the baby. And he didn't even ask me this morning, he just told me he was going. So I told him he could take it but that we need to talk about him using my car because I am stranded when he takes it for long periods of time. I guess part of me doesn't want him taking off in it. I pay for it, I maintain it, I even paid a parking ticket he got in it because it's my responsibility and I don't want to wait and wonder if he would pay it. 
Am I wrong for setting guidelines?