Knowledge · by Simon Willison (pattern) / GitHub
Git scraping
Keep a full history of how any web page changes.
link → data, series · link → data
- Way in
- Official APISign up yourself, api key
- To start
- FreeGitHub Free
- Runs
- On your Mac
- Checked
- Nov 3, 2026Prices as of that day
What it can do 2
Snapshot any JSON/HTML endpoint on a cron and keep full change historyActions cron minimum ~5 min and often delayed; scheduled workflows pause after 60 days of repo inactivitylink → data, seriesfree (public repos: Actions minutes free; private: 2,000 min/mo free)
Read an existing git-scraped dataset (e.g. CISA KEV mirror, CVE list, cloud IP ranges)GitHub raw CDN; API 60 req/h anonymous, 5,000 with tokenlink → datafree
Prices and limits as of Nov 3, 2026, from Simon Willison (pattern) / GitHub’s own pages. They change; check before you rely on them.Not affiliated with Simon Willison (pattern) / GitHub. Not advice.Something wrong?