A crawler has two high level options: parse the page, or render the page. Most o...

		dvanduzer on Oct 18, 2019 \| parent \| context \| favorite \| on: The Wayback Machine: Fighting Digital Extinction i... A crawler has two high level options: parse the page, or render the page. Most of our parser-based crawling is done by Heritrix (crawler.archive.org) and most of our render-based crawling is done by a proxy-based recorder similar to what you theorize (https://github.com/internetarchive/brozzler).

Thanks for sharing. That lets me sleep a bit easier.