To soon to give up?

My husband and I have been together for 7 years and only married two years on the 14th. I am currently 7 months pregnant with our third child. Just before I found out I was pregnant in June he told me he didn't know if he still loved me and he wanted to leave. During this time he started talking to a girl that he worked with. We didnt speak for two weeks and then he came back saying that he wanted to work things out and he was being stupid. He and the girl both swore they only talked. The rest of July and August go Good and then on September 1st he tells me the same thing. The whole month of September we barely speak and he keeps saying he is going to move out and doesn't. I finally write him a long email telling him how we will split bills and things and that he needed to be out of the house by the following Monday which was the 1st of october. He wound up moving out and then coming back a few days later saying he was sorry for everything and admitted to having a relationship with this girl from work the whole month of September... I took him back and We were trying to work on things. At the end of October the same thing happens... he moves put comes back within a week. now he is doing It again saying he wants to be alone and needs time to himself swearing its not because of me or anyone else (no I don't believe that he is not talking to her I'm not totally stupid). Anyways I told him that if this is what he wants this time I am filing for divorce and it is done. I am starting to fill out the paperwork and just feel so wired about it like I am giving up on us. His parents say that I am jumping the gun and we have been together since our senior year of high school. We are young and things are stressful and he just doesn't know how to handle it. Part of me feels like I need to be there for him to help him figure things out and be there through the worse. but then I think about what I have find through these whole last 7 months and pretty much dealt with this pregnancy alone and finished two quarters of school. I keep thinking that he is just expecting me to be here when he is done and I feel that I am worth more than being a second choice. Am I being selfish and giving up to early or is it time for me to let go and take care of my kids and myself.?