Web and news · by Common Crawl Foundation
Common Crawl
Grab the text of billions of web pages, archived monthly.
→ page, text · link → link, data · → page
- Way in
- Official APINo sign-up, no key
- To start
- Freefully free
- Runs
- In the cloudAlso through aws-open-data
- Checked
- Nov 3, 2026Prices as of that day
What it can do 3
Download raw HTML or extracted text for billions of pagesHundreds of TB per crawl; throttled under heavy load → page, textfree (S3 us-east-1; HTTPS via data.commoncrawl.org)
Look up all captures of a URL/domain in a crawl (CDX index)index.commoncrawl.org is rate-limited; use columnar index on S3 with Athena/DuckDB for scalelink → link, datafree
Pull daily news articles (CC-NEWS) → pagefree
Prices and limits as of Nov 3, 2026, from Common Crawl Foundation’s own pages. They change; check before you rely on them.Not affiliated with Common Crawl Foundation. Not advice.Something wrong?