Should I even try?

So started talking to this guy I had previous relations with but the thing is the reason we took a "break" was because he stopped talking to me out of the blue back in November. I was so pissed at the time because I really thought he might even be the one. So I texted him really late about 2 weeks ago and wished him a late birthday I didn't think he would reply since I knew he worked late but he did and he actually ended up calling me. So we texted for some days and he kept asking me to take him back. I told him I would think about it because I didn't want him doing the same thing again. He agreed and said he wouldn't do it again and he knew he messed up. Since I honestly really missed how great he was in bed I eventually took him "back" but I told him he better not mess it up again. So now we've been back at it again for maybe not even a week but mainly just though sexts but he's starting to take forever to reply, even a day and when he does its one worded texts. I even asked him the other day if I was being a bother and he said no. So I brushed it off and tried texting him again but now he hasn't replied at all and its been almost 2 days. This is what he was doing towards the end last time. Should I confront him? I feel like he wouldn't care. What should I say? I really don't want break him off but I feel like it's what I have to do. Like why would he do this if he practically begged for me to take him back?