should I break up with him?

Okay so I'm a junior in high school and I just recently got back with an ex boyfriend from sophomore year. During sophomore year we had been together for eight months and I lost it to him :/. This year when we got back together he said he truly wanted to be with me because I deserved to be treated better. Now that we've been back together for a couple of months he seems to almost ignore me. We will walk to our seventh period class together but he won't do little things like hold my hand, kiss me, or hug me. Which I know sounds stupid but we barely get to see each other in or out of school I would figure he would take any chance he would get to quote, acknowledge me. Now I would talk to him about it but he doesn't like talking about things like that. If we do talk about it he gets mad. And I don't like it when people get mad at/around me. What should I do?