Renaissance Labs

Atlas / North America

Nonprofit · San Francisco, United States

MLCommons

A vendor-neutral consortium that builds and publishes the MLPerf benchmarks, open datasets and AI safety measurements used across the industry.

evaluationdatasetsbenchmarks

San Francisco, United States

Type
Nonprofit
Based in
San Francisco, United States
Key papers
2
Featured projects
7

About

MLCommons grew out of MLPerf, a benchmarking effort begun in 2018 after engineers and researchers from Baidu, Google, Harvard, Stanford and UC Berkeley began meeting to agree on common ways to measure machine learning performance. The MLCommons association was launched in December 2020 to give that work, and related dataset and tooling projects, a permanent home.

Its working groups publish benchmark suites for training, inference, high-performance computing, mobile and tiny devices, together with reference implementations on GitHub. Datasets such as People's Speech and tooling such as MLCube are released openly.

Since 2023 an AI Risk and Reliability group has built safety measurement, which produced the AILuminate benchmark for large language models in December 2024. Member companies, universities and non-profits fund and govern the consortium, and its results are widely cited by hardware vendors and cloud providers.

Mission and ambitions

MLCommons states its mission as standardizing AI measurement so that buyers, builders and policymakers can make informed decisions about accuracy, safety, speed and efficiency.

Known for

  1. 01

    MLPerf training and inference benchmarks

  2. 02

    People's Speech open speech dataset

  3. 03

    AILuminate LLM safety benchmark

  4. 04

    MLCube portable ML packaging

Milestones

  1. 2018

    MLPerf Training launched

    Working group announced the first MLPerf Training benchmark in May after early meetings among industry and academic engineers.

  2. 2019

    MLPerf Inference launched

    The inference benchmark was announced in June, with first results published in November.

  3. 2019

    MLCube created

    A packaging standard for portable machine learning workloads was introduced in November.

  4. 2020

    MLCommons association launched

    The consortium was formally launched on 3 December, absorbing MLPerf and the People's Speech effort.

  5. 2023

    AI Risk and Reliability group formed

    A working group on AI safety measurement began in November.

  6. 2024

    AILuminate v1.0 released

    The first benchmark for assessing the safety of general-purpose chat models was released on 4 December in English.

Open source

GitHubgithub.com/mlcommons

Research papers

Notable releases

Openness

Open code and datasets; licences vary.

People

Listed from the lab's own pages and public profiles.

Contact and community

Contact details and links are on the lab's website (https://mlcommons.org); code questions go through issues on their GitHub organisation.

Sources

Compiled from the lab's public pages, Hugging Face and GitHub. Last checked 2026-11-17. Something wrong or out of date? Tell us on Discord.

Labs nearby