Should I be worried?

My husband is his ex's carer I've always had an issue with this, since we first got together she has always tried stuff with him even tho she's in a relationship herself, they have kids together but she walked out on them and signed them over to him. I've always asked why is he still her carer and he said because she needs him, even tho she's falsely claiming benefits. Monday his meant to be going round to see someone with her, her partner will be at work so it's just going to be them two. Am I in the wrong for being hurt and upset, it feels like his choosing her.