Federated Wiki at One

In one year we've converted vague notions of a new wiki into a working software platform with sufficient plugins to meet a demanding open-data application. Here we reflect on this experience from a software development perspective.

I noticed early in my experience with wiki that I had inadvertently concentrated power when the web's architecture was intended to distribute it. Correcting this mistake remained a long term goal.

Folk Memory was conceived while attending a distributed computing conference. I thought, the natural properties of a distributed system were features, not flaws to be overcome. conference

Sister Sites joined namespaces among cooperating wiki sites. My original wiki spawned several more. I asked that their hosts participate as a Sister Site if they took content from mine. I promoted this idea at the first WikiSym. conference

Smallest Federated Wiki was imagined as a weekend project to explore the Folk Memory concept of "trails". Soon after, I proposed continuing this barely started effort within my responsibilities as an open-data fellow. camp

As a Nike fellow, I set out to develop SFW as an open-data platform. Under the influence of numerous interviews with potential users I began to create pilot sites. These guided ongoing development while demonstrating concepts I felt to be important.

Community Garden provided data that could be collected and streamed off premise without attracting too much attention. I suggested that data-streams be approved for release, not data-sets.

Sensor Server data was converted from one flat file format to another using a perl program running as a cron job. The latter was the json expected by the emerging SFW client. I updated it every 5 minutes but argued that every 3 months would still be "real time" if it were similarly automated and didn't require additional review.

Factories included historical and geographical data. This seemed ideal for feeding visualizations cast as d3.js animations. I explored force layout on a base map as a "usefully inaccurate" representation of densely located sites. I expected the enter/exit "join" mechanism of d3.js to show dramatically when reporting methodology lead to a jump in reported locations.

Batch Jobs managed via relational database and hand-made visio diagrams became another target of d3.js force layout. I sorted 44,000 records into 12,000 pages and then organized these along corporate organizational boundaries. I argued that each department to manage their own jobs and that these could be "forked" back and forth with corporate IT.

Material Sustainability Index was a soon-to-be-released dataset that had been reviewed and found too hard to understand and change. I suggested federated wiki could address both issues.

Pattern Efficiency as measured by image analysis performed client-side in the wiki.

While SFW continued as an open-source project, executive support was sought and received for delivering material data in that medium. The workbook and accompanying pdf were revised. Wiki supplemented this as either a read-only version that could explain the data and methods or a read-write version that could support long-term maintenance of the dataset.

Phrase Analysis of existing documentation showed terms related to measurement, framing, indexing, policy and evolution. These were assembled into an influence hierarchy which guided further design.

Information Architecture at the page level (modeled with cards) showed data flowing left to right and ending in a visualization.

Representational Plugins for tabular data and its rendering in scatter charts and radar diagrams.

Workbook Export from excel to checkers, parsers and finally page builders.

Wiki Import interpreting workbook conventions and translating them to left to right, top to bottom data flow.

Computational Plugins for forward dataflow in methods and reverse flow in rollups.

Workbook Analysis using Parselets and Graphviz. svg

Development Methods based on mockup, code generation and test agains excel as oracle. Enhanced checks and traceability.

Trouble Sources from inventory, oracle, manipulation of workbooks; case, code and value variation of data; size and complexity of builder; asynchronous evaluation in client code.