We've created and run a federation wide scrape without instrumentation to tell us what it is doing. We've run some preliminary tallies and now repeat them to understand what the program has done overnight.
We report the page count and domain name of sites found to be online and reporting pages in their sitemaps. page 
We fetch once per second so we can predict runtime.
> Search Results 51 217 / ((3600 units) / hour) = 14.2269444 hours
From file system creation dates we see we started at Feb 21 20:49, about 3 hours to midnight. we now estimate completion at 14-3 = 11 am today.
So now numbers at 6 am, 9/14 = 64% done.
We refresh our counts of sites visited and welcome pages found to compute 126/213 = 59% of sites edit welcome.
ls data/*/welcome-visitors.json | wc
We count unique slugs over all sites.
11 indie-web-camp.json 11 json-schema.json 12 incremental-paragraphs.json 13 recent-submissions.json 14 chorus-of-voices.json 16 local-editing.json 17 smallest-federated-wiki.json 20 how-to-wiki.json 29 scratch.json 58 ward-cunningham.json 114 welcome-visitors.json
We count unique sites named by fork or reference.
25 tim.au.fedwikihappening.net 26 fed.coevolving.com 26 new.fed.wiki.org 26 nmsi.fed.wiki.org 27 machines.hapgood.net 30 design.fed.wiki.org 30 plugins.fed.wiki.org 34 jon.sf.fedwikihappening.net 35 frances.uk.fedwikihappening.net 35 h2.ward.asia.wiki.org 36 journal14.hapgood.net:3000 37 wiki-paul90.rhcloud.com 39 clive.tries.fed.wiki 41 goals.pod.rodwell.me 42 sfw.c2.com 42 video.fed.wiki.org 43 kate.au.fedwikihappening.net 44 maha.uk.fedwikihappening.net 47 wiki.dbbs.co 50 code.fed.wiki.org 50 localhost:3000 59 future.fedwiki.org 76 fed.wiki.org 86 journal.hapgood.net 98 found.ward.bay.wiki.org 101 ward.bay.wiki.org 174 ward.asia.wiki.org 197 forage.ward.fed.wiki.org 319 ward.fed.wiki.org
Is this program still working as expected? Let's look at what it is doing right now and see if it is producing one new file every second. Yes, five new files in five seconds.
touch now sleep 5 find data -newer now -name '*.json' data/habib.outlandish.academy/platform-earth-activity.json data/progressivedemocracy.cc/socrates-hated-democracy.json data/hosting.permanent.wiki/controlling-nginx.json data/thought.garden/cartesian-mindset.json data/518885.shep.hill.fed.wiki.org/ademar-aguiar.json