Federation Scraper

Here we supervise the ongoing scrape of the wiki federation. We invision this as cooperating loops where sitemap fetches lead to page fetches and these lead to more sitemap fetches.

See Stepping the Async Scrape

While developing this technology we focus first on a nested loop. We have several versions of this where we explore different instrumentation strategies.

Mock Computation

Start or Stop the Scrape

This page has been forked from a federation outpost that wraps a substantial, long-running, and data intense application with a wiki friendly interfaces for operating and documenting work done there.

See Federation Friendly Outposts

The implementation takes a fresh look at all implementation decisions. This is not our first step back, but it is the rethink that has us most excited today.

The usual federation navigation mechanisms may not be available yet. But more than you might expect works. Try the mock computation, a triple loop running in a San Francisco datacenter. lineup github