Brother's Kids

So I have a brother who is married but separated...his wife ended up moving a year ago to Oklahoma...he was in the beginning sending money to help with his kids but has stopped for a long while...now all he does is drink n talk smack to people when he is drunk..we don't talk at all anymore because I've told him over n over to stop drinking n be a father to his kids...now it's been alittle over a year n I knew this summer I wanted my neices(my nephew is barely 1 n I didn't want to take him from his mom for over a month)...my so knows how important my family is to me..I do all I can for my neices n nephews...I even adopted three of my nephews because my sister wasn't fit...so this past Friday my so was like u want to go get ur neices...n I'm like of course...so we left n made the 17 hour trip one way(I'm also 18 weeks pregnant)...we take turns driving there n back n make it back within 48 hours..the trip alone cost us 350 but sooo worth it...now all of a sudden I'm getting calls n texts from my brother n he wants to see his kids...I don't want to be mean but I want to tell him no...if he wanted to be a father he should of sent his kids money for their needs..I sometimes had to send money so my nephew could have diapers n wipes or my neices could have the luxury of eating pizza one evening...he just can't decide when he wants in their life n show responsibility when it's convenient for him...am I wrong? Should I give him a chance? I mean I made the choice to spend that money n take that time n get them...shouldn't I be able to tell him no n there r consequences to ur actions as a father????