Boyfriend problems

This has always been our downfall in our relationship. For some reason I am extremely defensive around him. I don't know what caused it from the start, but he likes to make stupid jokes and I get offended so easily. I'm such a sensitive soul and tend to take things personally and to the heart. He knows this very well but every time I get upset over a comment or a joke, he'll become extremely irritated and frustrated. How do I fix being so sensitive? I can't ask him to change his sense of humor. I just feel like a big baby... he said I'm driving him away by shutting down and becoming quiet when he makes a joke that upsets me. I shut down because I don't know how to handle his frustration and I freeze up because I'm scared of making things worse. This is a pattern and happens frequently. What am I doing wrong?