Renaissance Labs

Atlas / North America

Nonprofit · Beverly Hills, United States

Common Crawl Foundation

A nonprofit that has published free, open web crawl archives since 2008, forming a foundation dataset for language model research.

datasetsweb crawling

Beverly Hills, United States

Founded
2007
Type
Nonprofit
Based in
Beverly Hills, United States
Featured projects
6

About

Common Crawl is a 501(c)(3) nonprofit founded in 2007 by Gil Elbaz, who had earlier co-founded Applied Semantics. Its premise was that web-scale crawl data should be available to researchers who lack the resources to run their own crawling infrastructure.

The foundation publishes monthly crawl archives, each typically containing more than two billion pages, in WARC, WAT and WET formats. It also releases URL and CDXJ indexes and host-level and domain-level web graphs. The data is hosted on Amazon Web Services and can be downloaded free of charge.

A small staff of engineers and researchers maintains the crawler and the datasets, supported by a board and an advisory group. The archives serve academic researchers, startups and the teams that build language models.

Mission and ambitions

Common Crawl aims to keep a large, current copy of the public web freely available so that anyone can analyze it. It treats open access to crawl data as a way to widen who can do research on the web and on language technology.

Known for

  1. 01

    Open monthly web crawl archives

  2. 02

    More than 10 petabytes of crawled data since 2008

  3. 03

    Web graphs with PageRank and harmonic centrality

  4. 04

    A common source of training text for language models

Milestones

  1. 2007

    Foundation established

    Gil Elbaz founded the nonprofit to publish crawl data for public use.

  2. 2008

    First archive crawl

    Crawl archives were made available publicly beginning in 2008.

  3. 2020

    Web graph releases

    Host-level and domain-level web graphs with Harmonic Centrality and PageRank values are published alongside the crawls.

Open source

Notable releases

Openness

Data free to use; code open source.

People

Listed from the lab's own pages and public profiles.

Contact and community

Contact details and links are on the lab's website (https://commoncrawl.org); code questions go through issues on their GitHub organisation.

Public Discord linked from the lab's own website for community discussion and announcements.

Sources

Compiled from the lab's public pages, Hugging Face and GitHub. Last checked 2026-11-17. Something wrong or out of date? Tell us on Discord.

Labs nearby