Boyfriend thinks I intentionally hurt him

A few months ago my boyfriend and i were joking about something and i playfully went to smack his arm but the way his hand was positioned made it so that i smacked his thumb. He has said multiple times that it hurts and causes him some pain and discomfort. A few weeks ago he was at work and caught a box in a weird way to where i caused his thumb to hurt even more and he blames me for it. He always said "you messed up my thumb" in a way where he thinks i had the intentions of messing up his thumb and brings it up constantly saying the same shit about how i fucked up his thumb and how he cant really use it. I have told him to go to the doctor but he doesnt. This all sounds so silly and IT IS. im just so annoyed with the fact that he believes i did it on purpose! I apologize everytime he brings it up when he cant use it and i dont know what to do! Am i supposed to baby him and kiss his ass?? What does he fucking want from me to constantly bring it up!! Im so annoyed. I keep saying to him "its not like i said in my head if he positions his hand this way, maybe i should playfully smack him and fuck up his hand." He acts like a fucking child and im so tired of it.