He's always cancelling our plans!

My boyfriend and i have been together for 4 years and have a 7 month old daughter. We dont live together as of yet so me and my daughter would travel to his and stay some nights. Well its his birthday today and for days he has been saying how we are all going out for dinner and spending some quality time together. 2 hours before we are due to leave he says hes going out with his friend for a drink and wont be back until later tonight. Normally id just let it go but hes been with his friend nearly all week and has only seen me and his baby for about 4 hours since sunday. Am i wrong for being angry about this? He recently lost his job last week so he has been at home and inviting his friend round to chill with him. He always cancels plans with me a few hours before to hang with friends but like i said i try to let it go because he needs his guy time.