Husband vs BD

So my husband doesn't necessarily care for my kids dad (we had a rough transition into a blended family nothing physical). My kids dad and I have 2 kids under 4. Their dad takes them every weekend faithfully per our visitiation agreement. My older daughters birthday is sunday so on dads day. I could ask for them but he would want to go not drop them off. I want to be able to ride the slides with my daughter and husband as well as enjoy some time with the babies and you know co-exist.

My question is should just invite him and see how it goes or not invite him or her sibling to the party but my husband don't care for him. The sibling have missed her party last year for this reason.