Husband leaving for our first father's day

I just had our daughter last month and I wanted us to spend the whole day together for our first father's day but my husband wants to split the day in half to spend with his ex step son. He's been in his ex step sons life since he was 15 months old and he's 14 now. He only found out a few years ago that my husband isn't his biological father but my husband promised no matter what happened between him and his ex wife he would always be in his life. He is wanting to leave at like 4 to take his ex step son out for dinner. I understand what I was getting into when I married him but shouldn't he want to spend our first father's day with his REAL daughter. His ex step son got plenty of father's days and we already see him all the time. I don't want to share this day with him.