advice please?????

Ar

So my boyfriend has pretty much raised himself, his mother left when he was young due to her other childs death . they blamed him , his father continuously beat him , it just recently stopped when he went to prison . My boyfriend has lived on his own since he was 14 . he knows how to pay his bills, how to keep food in the house , how to work for his money, how to save and to pretty much just be a man but because of his past , without his mother and his father he comes with issues . And i totally understand that he needs time to heal from everything but with his emotional state its hard to not think of him as a two year old throwing fits because he doesn't understand why his fish is gone .

He and I are both working on his self esteem and how he views everything. He has a great heart , he really does . he comes from a wealthy family on both ends so he always tries to give to families in need , down to where hed give the clothes on we wore on his back . But he doesn't trust any blood relatives , and close friends to his families but he has no problem with trying to move in with the neighbors . he tried going home with everyone, which i try and understand because thats how he was raised being passed around to homes . ( my mom is a foster parent so i see this a lot so i try not to judge him ) i really do care for him , he is my first love and honestly i know eventually we will part but I'm trying to better him before that day comes so he doesn't get hurt anymore than what he is . Now everytime we argue he makes himself the victim even when hes wrong and i do give in every now and then . But my question for all of you is how do i respond when he sends me this ? I tell him that he isn't worthless and I explain to him why it is he's in the wrong but its like I'm dealing with my autistic brother and it annoys me sometimes . I'm not gonna lie he annoys me when he brings this on because its him constantly repeating himself and I'm running out of ways to get him to speak his words instead of trying to get me not to leave him . I get this may seem like a toxic relationship but honestly we have a million and one strong points so please do not tell me that this is toxic .