Need some advice but judgement

My SO and I have been together gong on 6 yrs now we have 2 children together a 3 year old girl and a 9 month old son. I have been faithful to him even when he has not up until 3 years ago now. Me and a co-worker have been having an "affair" (neither one of us is married to our SO) we were talking before me and my SO got together. Then 3 yrs ago we started talking to each other just as friends nothing more we both had a small child at the time (our daughters are the same age just 3 months apart) well my SO was cheating with a girl he worked with on second shift. He wouldn't give me no attention and we barely had sex once every 3 or 4 months if I was lucky. He claims he has a low sex drive for a man with 4 other children before he met me (we were both single when we met) I find that hard to believe so me and the co-worker started a sexual relationship and he is everything I could want in a lover or a man for that matter. My SO found out about our 2 yr sexual relationship back in July of this past year and he said he would try and change but he hasn't. We don't have sex, we don't sleep in the same room together I sleep in the bedroom with my children and he sleeps in the living room, he'd rather watch porn than have sex with me. He rather play his video game than tend to the kids while I'm cooking or putting laundry away. I don't have the convience of being a stay at home mom I work 8-10 hrs a day come home and take care of things there my day don't end till my kids are in bed and I finally lay down. I pay all the bills he barley brings home enough to cover the weekly daycare bill because he's on child support for 3 of his 4 children (my 2 make 6). My coworker is unhappy with his home situation so we basically stuck because of our kids but we both have discovered we have very strong feelings for each other and have the same situation going on as far as sex and sleeping situation. We both feel our current relationships are at their end but neither one are sure of how to take this next step and now there could possibly be a new baby (co-worker and I). Just looking for thoughts or advice on the situation.