Benchmark models independently on identical hardware and publish a simple comparable efficiency label

Location

Paris, France48.8566, 2.3522

Description

AI Energy Score, launched in February 2025 at the Paris AI Action Summit by Hugging Face, Salesforce, Cohere and Carnegie Mellon University, benchmarks models on standardised NVIDIA H100 hardware across 10 tasks and assigns GPU energy into a 1-to-5 star rating. Infrastructure includes a public leaderboard (166+ models at launch), a submission portal, a label generator, and a secure Docker sandbox for closed models. A v2 refresh in December 2025 added reasoning models. Salesforce became the first developer to disclose proprietary-model energy and now auto-generates model cards with it. Most other closed providers have not submitted production models, leaving comparison ranges incomplete. The v2 data found that of 14 comparable models, 8 used equal or more energy than their February counterparts.

Metrics

3

Models on the public leaderboard at launch0166+models

Tasks benchmarked per model10tasks

Comparable models showing equal or greater energy use in v2 vs. February baseline8 of 14models

Lessons learned

Voluntary disclosure has so far reached one major closed developer; replicators should plan for incomplete coverage and consider how to incentivise or require submission from closed-model providers.
Fixing hardware (all models on H100) is what makes comparisons fair, at the cost of not reflecting any specific production deployment — a trade-off to document explicitly for users.
Empirically, newer models are not uniformly more efficient; an ongoing public benchmark is necessary rather than assuming progress over time.
Independent benchmarking creates procurement pressure that pure estimation cannot, but it complements rather than replaces disclosure mandates or regulation.

Sources

3

Links

2

Documented Jul 13, 2026

Arnaud Gissinger