Time to give up? Divorce?

I just recently quit my job to stay at home because I am 6 months pregnant. My husband & I have been together 4 years. Married 2. He has never kept a steady job & I have always been the main provider. I have a 6 year old daughter with another man. I have always been the responsible one. He promised he would take care of the bills & I pretty much told myself it was sink or swim. He could do it now & if not he never would. Well we're barely floating. Not only is he still not paying the bills (I am) but he's pretty much proving to me more & more that I don't need him. I rarely cooked due to working over 52 hours a week. I decided to cook & have dinner ready when he got home one evening,,... Only for him to tell me he ate with his buddies after work. He claimed he felt so bad because he knew I was trying.. It's like when I shave my legs they don't get touched. When I tell him I feel deprived of affection & attention,,, he sleeps on the couch. I'm literally at that point I feel like I've done all I can do. I've supported us. I've tried to do the housewife thing. I have a huge fear of having to go thru raising a newborn alone but when is it enough?