I'm not the asshole parent..

My soon to be ex husband and I have not been together in a year. We have 4 children together they are 9,7,6 and 3. We've had a ton of issues neither one of us was perfect. When I finally decided to leave he took it very hard and turned to things I never thought he would the first 6 months he seen the kids maybe 3 times and no longer then 15 minutes. Over Christmas he went to jail for drug charges, he's out in bond and again seen the kids twice. Well he's "getting sober" hasn't used in 2 months. So I've allowed him to have phone calls with the kids, he's called every night for the past week. His birthday is Thursday, my kids are making him birthday cards at the moment. My family is pissed and keeps asking why I allow the kids to do these things she talk to him.. Well, I'm not going to be the asshole parent, my kids will develop their own opinion on him and as long as they are not getting exposed to his life style they have every right to do so. How do you tell a 6 year old no you can't make him a card.. my family really don't understand my view on this..