Web and news · by Common Crawl Foundation

Common Crawl Index

Look up copies of billions of pages saved each month.

link → data · data → page
Way in
Official APINo sign-up, no key
To start
Freefree
Runs
In the cloudAlso through aws-open-data
Checked
Nov 3, 2026Prices as of that day

What it can do 2

Look up captures of a URL/domain in a crawlshared index server; heavy users should query columnar index on S3/Athenalink → datafree
Fetch archived page bytes from WARC by byte rangedata → pagefree (HTTPS or S3 us-east-1)
Prices and limits as of Nov 3, 2026, from Common Crawl Foundation’s own pages. They change; check before you rely on them.Not affiliated with Common Crawl Foundation. Not advice.Something wrong?
Made by Jasper Burns

A short note when I ship something new, about once a month. One list for all my projects.