Knowledge · by Simon Willison (pattern) / GitHub

Git scraping

Keep a full history of how any web page changes.

link → data, series · link → data
Way in
Official APISign up yourself, api key
To start
FreeGitHub Free
Runs
On your Mac
Checked
Nov 3, 2026Prices as of that day

What it can do 2

Snapshot any JSON/HTML endpoint on a cron and keep full change historyActions cron minimum ~5 min and often delayed; scheduled workflows pause after 60 days of repo inactivitylink → data, seriesfree (public repos: Actions minutes free; private: 2,000 min/mo free)
Read an existing git-scraped dataset (e.g. CISA KEV mirror, CVE list, cloud IP ranges)GitHub raw CDN; API 60 req/h anonymous, 5,000 with tokenlink → datafree
Prices and limits as of Nov 3, 2026, from Simon Willison (pattern) / GitHub’s own pages. They change; check before you rely on them.Not affiliated with Simon Willison (pattern) / GitHub. Not advice.Something wrong?
Made by Jasper Burns

A short note when I ship something new, about once a month. One list for all my projects.