Was I wrong...

My husband always picks petty arguments when he has days off from work and we end up just doing our own things and not talking much for the whole day. We have been having problems in our marriage and have talked about divorce. He always comes back around when he wants sex. He can act normal at times and then he's right back on the divorce train when it suits him. I'm 24 weeks pregnant right now and we are raising 3 other children together as well. He is the only one working and today he talked about going to get beer and the get something to eat just or himself. I asked "Do we have the money for that?" An he said "You mean do I have the money for that?" I just turned away. He got the beer and came back. We hadn't said a word to each other when he walks into our bedroom and said "I need your mouth" an I told him no! He went on a rant about it and brought up divorce. I'm not just here for sex! If he wants a divorce then I will not be having sex any more! I hate that he just thinks that he can treat me this way...am I wrong?