crawl

crawl is a simple web crawler that is local to one domain. It’s using goroutines to parallelize things.

The generated sitemap is exceedingly simple, we do not keep track of a real graph, but rather just a shallow list of links for every page.