Husband won’t let me live freely

My husband is obviously not in love with me and neither I am with him. However we have a child and we both financially depend on each other because we share many assets. I know he talks to other women because I’ve caught him and I am also talking to other men. He knows about the men I’m talking to and doesn’t care but he tells me we are not going to get divorced if I go live with one of the men I’m talking to because they aren’t wealthy. He says he wants me and my son to live a “better life” and that we deserve to be happy. He doesn’t want “poor man” raising my child. I understand his concerns and I don’t plan on living a poor lifestyle because I make good money. If we are separated from each other I’m not jumping into a new relationship right away and will just focus on myself and son, but he’s afraid I’ll end up living with one of the “poor guys.” & he said no Hispanics (he is Hispanic and immigrant himself. I’m white) Is he right or wrong? Jealousy? I am confused by the way he thinks.