Is it even worth it?

So lately I have just been super irritated with my boyfriend. Literally we walks into the room and I'm just pissed. We have a son together and a daughter on the way. I have a super high sex drive and we haven't had sex pretty much since I got pregnant. But even before that our sex life just went to shit after the delivery of our son I don't know if it changed his mind set about me or what but we had a good sex life even when I was pregnant with my son. I'm just so irritated, and then I caught him talking to another girl last month saying he would go visit her when he got out of school. (He in college) but his classes are until 10 pm and I found out because I went through his phone. (I know total invasion of privacy but I knew something was up) when I bring up that our relationship is just going to shit he always says it's my fault and then questions me about if the kids are his. And he's been the only guy I've been with for four years. I just don't know what to do anymore. I love him because he is my children's dad but I just don't feel like I am in love with him.