Ex won’t let me claim son on taxes

So my ex and I have a 4 year old son together and we broke up two years ago. He has had him on his medical insurance plan so I already let him claim him. Well not I have a newborn with my new bf and I now want to add my 4 year old to my medical insurance because I have a “family” plan that I can add up to 3 children for the same price, and this way I can claim our 4 year old. Now my ex says no way even though we said before, whoever pays insurance can claim him. What can I do? I don’t think it’s fair. Fair would be every other year and now he says nope! Advice?