Web and news · by Common Crawl Foundation

Common Crawl

Grab the text of billions of web pages, archived monthly.

→ page, text · link → link, data · → page
Way in
Official APINo sign-up, no key
To start
Freefully free
Runs
In the cloudAlso through aws-open-data
Checked
Nov 3, 2026Prices as of that day

What it can do 3

Download raw HTML or extracted text for billions of pagesHundreds of TB per crawl; throttled under heavy load → page, textfree (S3 us-east-1; HTTPS via data.commoncrawl.org)
Look up all captures of a URL/domain in a crawl (CDX index)index.commoncrawl.org is rate-limited; use columnar index on S3 with Athena/DuckDB for scalelink → link, datafree
Pull daily news articles (CC-NEWS) → pagefree
Prices and limits as of Nov 3, 2026, from Common Crawl Foundation’s own pages. They change; check before you rely on them.Not affiliated with Common Crawl Foundation. Not advice.Something wrong?
Made by Jasper Burns

A short note when I ship something new, about once a month. One list for all my projects.