Should I trust him with baby?

So my boyfriend/baby's father has never been alone with our baby. He's 3 and a half months old. He's barley bonded with him. It seems like he is not very interested, even though he tells me he raised two other children and was very involved according to him. I don't trust him with him normally because he drinks all day long. And by the evening he's already halfway to drunk. He gets annoyed when our son wakes up and is fussy. He told me he wanted to flick him the other morning. Well I have an appointment tomorrow morning at 7 and he wants me to leave him with him. I really don't feel comfortable doing it but he's making me feel bad because he is his father. What do you guys think? And whenever he is holding him and he gets fussy he hands him back to me right away.