dat

Dat is a version-controlled, decentralized data sync tool designed to improve collaboration between data people and data systems - github

The dat module is designed with a small-core philosophy. It defines an API for reading, writing and syncing datasets and is implemented using Node.js.

See also Wikidata for Research

This release is the first step towards our goal of creating a streaming interface between every database or file storage backend in the world. We are trying to solve hard problems the right way. This is a process that takes a lot of time. github

In the future we would also like to work on a way to easily host and share datasets online. We envision a sort of data package registry, similar to npmjs.org, but designed with datasets in mind. This kind of project could also eventually turn into a sort of "GitHub for data".

We also want to hook dat up to P2P networks, so that we can make downloads faster but also so that datasets become more permanent. Dat advisor Juan Benet is now working on IPFS, which we are excited to hook up to dat when it is ready.

Certain datasets are simply too large to share, so we also expect to work on a distributed computation layer on top of dat in the future (similar to the ESGF project).

Internally dat has two kinds of data storage: tabular and blob. The default tabular data store is LevelDB and the default blob store stores files on a content-addressable blob store. Both of these default backends can be swapped out for other backends.

Show current status, including row count, file count, last updated: