Should I or shouldn't I?

We spoke for about a month and in that time sort of stopped a couple times because of various things. The last time we stopped talking it was because I panicked I guess about getting hurt. But I kept thinking about him and reached out apologising. It didn't go any further but about half a week after that he messaged me thanking me for the time we spent talking because I showed him how that he can be treated right. We spoke a bit on the phone last night and the idea about meeting up was thrown about. Thing is because it's been a couple weeks since we last spoke properly I'd be a bit concerned that things aren't the same or that deep down I'd be worried his feelings changed because I'm the one who called it off. He told me his feelings haven't changed. But I've been feeling pretty vulnerable lately in terms of my mental health, I've felt very down and sad, he does make me happy. But I'd be worried if we went on a date and he rejected me (this has happened before to me) it would crush me because I feel I'm not in a place to deal with rejection at the moment. I'm going away next week for my birthday and I don't want to feel down I guess but at the same time he's a lovely guy and if I didn't go on a date with him I'd be thinking "what if" which I don't know if that'll be worse. Has anyone been in a similar position? I don't know whether I'm best on my own for a while (thing is I'm completely alone, I have nobody and I loved talking to him cause he was so similar to me) and heal my mind slowly then go back to the dating scene. Or give it a go with him this weekend. I haven't spoken to anyone else like him before, we got on well and have good phone conversations but I don't wanna harm my mental state even more to avoid being on my own. Or say to him no to a date and constantly wonder what could've been.