Elo ratings beyond arena-style evaluations

Elo ratings were born in chess halls, where every move is visible and every opponent is human. Transplant that into AI evaluation, and you get leaderboards that feel objective but hide a mess of variance, selection bias, and cultural assumptions. Cohere's latest post doesn't just tweak the formula—it forces us to ask if the arena itself is the right metaphor. Here's why this matters: evaluation is the silent governor of AI progress. If our metrics reward style over substance, or confuse popularity with capability, then every model trained to rank high is being aimed at a distorted