How to break up with my boyfriend

My boyfriend and i are in a long distance relationship. We live 2 hours away from each other and I really like him but his house is nasty and smells like cat pee. I’ve tried to clean up but the smell is awful it makes me not want to go there and I’ve told him that (I’m really allergic also). I mentioned the cats are an issue and he’s trying to do things to help but i can not take his house... I broke up with him last week but it hurt so i went back to him and it hurt him too which i didn’t wanna do. I just moved back to NJ(where he lives also) and just now getting my life together: finally have a job, I’m in school, and working on myself and my relationship with my family. He’s 25 (done with school regular job) I’m 20. I asked him if he’d be comfortable just coming here but he’s not happy with that. He’s not happy with how often he’ll see me, he’s not happy that he’s trying and it’s not enough. I’ve offered to help him clean and I’ve offered to help him get the smell out too, and his response was “it smells... and is uncomfortable... to YOU. You want it to make yourself more comfortable..” he says he’s sexually frustrated and hates our distance. I’m stressing... i hate going to his house.. I’m in school.. just got a job.. he wants to see me every weekend or 3/4 and i don’t know if it’s possible. i don’t know why i didn’t just stay broken up with him bc i don’t know how to do this w/o hurting him. It’s stressing me out. This time it won’t hurt me if we split but i don’t know how to break it to him. I’m fine with our distance, i don’t need to see him every weekend, I’m not sexually frustrated, but he’s not happy. We’re not right for each other how do i get out :(