Naps?

So this is a controversy in my household. My son is almost 2 1/2. We still give him naps. Usually he will sleep for 2 hours. But lately they have been getting to be later and later into the day. He doesn't get up till 8-9am, so he's not ready for a nap until 2-3pm, that means he will sleep until 4-5pm. So when that happens, 8:30-9pm comes along and he wants absolutely nothing to do with going to bed. It's the toughest part of the day. Putting him down to bed takes a good 30-45 mins. Now we have been slipping and letting him stay up till 9:30-10 which is throwing everything off. I tried shortening his naps but that doesn't help either. So maybe it's time to take away the naps all together? When we do that, he is usually falling asleep on the couch or ready for bed by 7:30 but could keep him up a little longer if we wanted. 
When did you know it was time for the naps to be over? What age?