Sad, hurt, confused

Recently found out that my husband developed an emotional connection with a co-worker. Things between them only went as far as talking and lasted all of about 2 weeks. So we discuss it, I am understanding about it. Things are good it seems. Fast forward to yesterday, I pull in the driveway, he meets me outside and we start talking as I am getting the baby out of the car. I asked how his day was and he says it was rough, been trying not to think about her. Her meaning the girl he started to get attached to. He was home on a vacation day and it seems like all he did was sit around and think about this chic. I really don't even know how to feel. I want him to be able to talk to me and tell me his feelings but there is a line. If what is going on in his mind is going to be hurtful maybe it shouldn't be one of the first things you say to your wife when she gets home from working all day. I was dealing with all of this. I was feeling positive about where we were headed and then I find out that he is sitting around all day thinking about some bitch because she is young and bubbly with blue eyes and blonde hair. She is 22, he is 36....they have nothing in common. He just likes the way she would look at him. I don't know what to do anymore, half of me wants to take my child and split, the other half wants to see if things can work themselves out. I just don't know how to go on like this isn't bothering me. When I made a comment about it last night we fought, he had the nerve to get mad at me because I got upset when he told me why his day was rough! I feel like I have every right to be mad and every right to want to rearrange this cute little bitches face!
EDIT: thank you all for your kind and encouraging words. While I do blame my husband for letting it happen, because he knows better. I also blame this chic because she knew he was married and has a family. Yes she is young but she is also old enough to know not to be a home wrecker. I have zero respect for women who knowingly make the decision to pursue a married man. At the time this took place we were fighting quite a bit but that is no excuse for his behavior. We both could stand for some counseling. I still love this man with everything I have and I am beyond grateful that he stopped it before it went to far. It speaks volumes to me because things could have easily gone too far. He told me that he didn't want to lose what he had. 
We actually got a babysitter last night and had a much needed night out. Where we talked and put everything out on the table. Things are on the mend. He apologized for what was said and agreed it could have been done differently. He told me that he wants me and our family. We made agreements to be better to one another and to make sure we make time for us. I do not blame myself for what happened but I do realize that if I am taking him for granted there is someone else out there that won't, so I need to be very conscious of how he feels too. The reasons that led up to this need to corrected on both ends. He also agreed that female co-workers would remain just that. I the that it had to come to his but on the other hand it may have been what was needed in order to fix us.