Web and news · by Common Crawl Foundation
Common Crawl Index
Look up copies of billions of pages saved each month.
link → data · data → page
- Way in
- Official APINo sign-up, no key
- To start
- Freefree
- Runs
- In the cloudAlso through aws-open-data
- Checked
- Nov 3, 2026Prices as of that day
What it can do 2
Look up captures of a URL/domain in a crawlshared index server; heavy users should query columnar index on S3/Athenalink → datafree
Fetch archived page bytes from WARC by byte rangedata → pagefree (HTTPS or S3 us-east-1)
Prices and limits as of Nov 3, 2026, from Common Crawl Foundation’s own pages. They change; check before you rely on them.Not affiliated with Common Crawl Foundation. Not advice.Something wrong?