Ex contacts me

Someone i dated in the past and slept with once contacted me recently asking to meet up for a 'catch up'. Im in a serious relationship now, is it rude that i bluntly told him i 'didnt want to because i would feel bad for my boyfriend' and i dont think its a good idea'? He freaked out obviously and got angry. But what would you do? If you didnt like someone at all, you had slept with them and you were in a serious relationship??????