
Nonprofit · Beverly Hills, United States
Common Crawl Foundation
A nonprofit that has published free, open web crawl archives since 2008, forming a foundation dataset for language model research.
Beverly Hills, United States
- Founded
- 2007
- Type
- Nonprofit
- Based in
- Beverly Hills, United States
- Featured projects
- 6
About
Common Crawl is a 501(c)(3) nonprofit founded in 2007 by Gil Elbaz, who had earlier co-founded Applied Semantics. Its premise was that web-scale crawl data should be available to researchers who lack the resources to run their own crawling infrastructure.
The foundation publishes monthly crawl archives, each typically containing more than two billion pages, in WARC, WAT and WET formats. It also releases URL and CDXJ indexes and host-level and domain-level web graphs. The data is hosted on Amazon Web Services and can be downloaded free of charge.
A small staff of engineers and researchers maintains the crawler and the datasets, supported by a board and an advisory group. The archives serve academic researchers, startups and the teams that build language models.
Mission and ambitionsCommon Crawl aims to keep a large, current copy of the public web freely available so that anyone can analyze it. It treats open access to crawl data as a way to widen who can do research on the web and on language technology.
Known for
- 01
Open monthly web crawl archives
- 02
More than 10 petabytes of crawled data since 2008
- 03
Web graphs with PageRank and harmonic centrality
- 04
A common source of training text for language models
Milestones
- 2007
Foundation established
Gil Elbaz founded the nonprofit to publish crawl data for public use.
- 2008
First archive crawl
Crawl archives were made available publicly beginning in 2008.
- 2020
Web graph releases
Host-level and domain-level web graphs with Harmonic Centrality and PageRank values are published alongside the crawls.
Open source
Crawl support library.
PySpark.
News crawl.
Statistics.
- CommonLIDdataset
Language ID.
- statisticsdataset
Statistics.
Notable releases
Openness
Data free to use; code open source.
People
- Rich SkrentaExecutive Director
- Gil ElbazFounder and Chairman
Founded Common Crawl in 2007.
- Greg LindahlChief Technology Officer
- Sebastian NagelDistinguished Engineer
- Thom VaughanPrincipal Engineer
- Pedro Ortiz SuarezPrincipal Research Scientist
Listed from the lab's own pages and public profiles.
Contact and community
Contact details and links are on the lab's website (https://commoncrawl.org); code questions go through issues on their GitHub organisation.
Public Discord linked from the lab's own website for community discussion and announcements.
Sources
Compiled from the lab's public pages, Hugging Face and GitHub. Last checked 2026-11-17. Something wrong or out of date? Tell us on Discord.



