Ughhhh he's so annoying!

C
So my boyfriend is a clean freak. And he's always getting upset with me about not being as crazy as he is about certain things. I am by no means dirty, but I've had to learn to keep our house in such a way or he would get upset. We even broke up over it st one point because he couldn't handle things not being done his way. The thing is, there are many ways in which I find him to be messy. He leaves empty jars in the fridge, our closets are packed and messy, but because these are important to him, they don't matter. Today he starts cleaning and that means he's pissed off. Cleaning seems to upset him so much that then I'm walking on eggshells because I'm afraid of setting him off more. So his mood is all pissy all day because of this, and he asks me if I make people take their shoes off at the door. I said yes, but that in some instances I might feel awkward saying something. Well this upsets him because his rule is to take shoes off and I should make everyone follow that rule. And I'm saying yeah, I get that and I generally would, but to me it's not the end of the world and I can't care about it as much as you. Not saying I won't enforce his rule, because I do, but I just can't get upset about it like he does. It's not worth it to me. 
So finally he says he's going for a walk and I was about to go out with our 4 month old daughter so I told him to take her out. He's never done this. I do 100% of the childcare with her and I could tell he didn't want to take her. He tried some lame ass excuse but I told him I could use a half hour to myself and sent him off. 
I realize it's hard to convey the backstory here to get the full picture of what this argument is about because I'm not even sure myself. I just never understand how these rules he makes up for our household can make him so upset with me. They aren't rules I always agree with so I just don't follow them the same way. I try to be respectful, but I'm a person in this house. I work, I pay bills, I have just as much a right to feel free here also and not get shat on every time he cleans the damn floors. 
Sorry for the rant, but can I get some perspective?