| 1 |
Background and motivation |
| 2 |
************************* |
| 3 |
|
| 4 |
Vision |
| 5 |
====== |
| 6 |
|
| 7 |
Data is at the core of science, and unobstructed access promotes scientific |
| 8 |
discovery through collaboration between data producers and consumers. The last |
| 9 |
years have seen dramatic improvements in availability of data resources for |
| 10 |
collaborative research, and new data providers are becoming available all the |
| 11 |
time. |
| 12 |
|
| 13 |
However, despite the increased availability of data, their accessibility is far |
| 14 |
from being optimal. Potential consumers of these public datasets have to |
| 15 |
manually browse various disconnected warehouses with heterogeneous interfaces. |
| 16 |
Once obtained, data is disconnected from its origin and data versioning is |
| 17 |
often ad-hoc or completely absent. If data consumers can be reliably informed |
| 18 |
about data updates at all, review of changes is difficult, and re-deployment is |
| 19 |
tedious and error-prone. This leads to wasteful friction caused by outdated or |
| 20 |
faulty data. |
| 21 |
|
| 22 |
The vision for this project is to transform the state of data-sharing and |
| 23 |
collaborative work by providing uniform access to available datasets -- |
| 24 |
independent of hosting solutions or authentication schemes -- with reliable |
| 25 |
versioning and versatile deployment logistics. This is achieved by means of a |
| 26 |
:term:`dataset` handle, a lightweight representation of a dataset |
| 27 |
that is capable of tracking the identity and location of a dataset's content as |
| 28 |
well as carry meta-data. Together with associated software tools, scientists |
| 29 |
are able to obtain, use, extend, and share datasets (or parts thereof) in a |
| 30 |
way that is traceable back to the original data producer and is therefore |
| 31 |
capable of establishing a strong connection between data consumers and the |
| 32 |
evolution of a dataset by future extension or error correction. |
| 33 |
|
| 34 |
Moreover, DataLad aims to provide all tools necessary to create and publish |
| 35 |
*data distributions* |---| an analog to software distributions or app-stores |
| 36 |
that provide logistics middleware for software deployment. Scientific |
| 37 |
communities can use these tools to gather, curate, and make publicly available |
| 38 |
specialized collections of datasets for specific research topics or data |
| 39 |
modalities. All of this is possible by leveraging existing data sharing |
| 40 |
platforms and institutional resources without the need for funding extra |
| 41 |
infrastructure of duplicate storage. Specifically, this project aims to provide |
| 42 |
a comprehensive, extensible data distribution for neuroscientific datasets that |
| 43 |
is kept up-to-date by an automated service. |
| 44 |
|
| 45 |
|
| 46 |
Technological foundation: git-annex |
| 47 |
=================================== |
| 48 |
|
| 49 |
The outlined task is not unique to the problem of data-sharing in science. |
| 50 |
Logistical challenges such as delivering data, long-term storage and archiving, |
| 51 |
identity tracking, and synchronization between multiple sites are rather |
| 52 |
common. Consequently, solutions have been developed in other contexts that can |
| 53 |
be adapted to benefit scientific data-sharing. |
| 54 |
|
| 55 |
The closest match is the software tool git-annex_. It combines the features of |
| 56 |
the distributed version control system (dVCS) Git_ |---| a technology that has |
| 57 |
revolutionized collaborative software development -- with versatile data access |
| 58 |
and delivery logistics. Git-annex was originally developed to address use cases |
| 59 |
such as managing a collection of family pictures at home. With git-annex, any |
| 60 |
family member can obtain an individual copy of such a picture library |---| the |
| 61 |
:term:`annex`. The annex in this example is essentially an image repository |
| 62 |
that presents individual pictures to users as files in a single directory |
| 63 |
structure, even though the actual image file contents may be distributed across |
| 64 |
multiple locations, including a home-server, cloud-storage, or even off-line |
| 65 |
media such as external hard-drives. |
| 66 |
|
| 67 |
Git-annex provides functionality to obtain file contents upon request and can |
| 68 |
prompt users to make particular storage devices available when needed (e.g. a |
| 69 |
backup hard-drive kept in a fire-proof compartment). Git-annex can also remove |
| 70 |
files from a local copy of that image repository, for example to free up space |
| 71 |
on a laptop, while ensuring a configurable level of data redundancy across all |
| 72 |
known storage locations. Lastly, git-annex is able to synchronize the content |
| 73 |
of multiple distributed copies of this image repository, for example in order |
| 74 |
to incorporate images added with the git-annex on the laptop of another family |
| 75 |
member. It is important to note that git-annex is agnostic of the actual file |
| 76 |
types and is not limited to images. |
| 77 |
|
| 78 |
We believe that the approach to data logistics taken by git-annex and the |
| 79 |
functionality it is currently providing are an ideal middleware for scientific |
| 80 |
data-sharing. Its data repository model :term:`annex` readily provides the |
| 81 |
majority of principal features needed for a dataset handle such as history |
| 82 |
recording, identity tracking, and item-based resource locators. Consequently, |
| 83 |
instead of a from-scratch development, required features, such as dedicated |
| 84 |
support for existing data-sharing portals and dataset meta-information, can be |
| 85 |
added to a working solution that is already in production for several years. |
| 86 |
As a result, DataLad focuses on the expansion of git-annex's functionality and |
| 87 |
the development of tools that build atop Git and git-annex and enable the |
| 88 |
creation, management, use, and publication of dataset handles and collections |
| 89 |
thereof. |
| 90 |
|
| 91 |
Objective |
| 92 |
========= |
| 93 |
|
| 94 |
Building atop git-annex, DataLad aims to provide a single, uniform interface to |
| 95 |
access data from various data-sharing initiatives and data providers, and |
| 96 |
functionality to create, deliver, update, and share datasets for individuals |
| 97 |
and portal maintainers. As a command-line tool, it provides an abstraction |
| 98 |
layer for the underlying Git-based middleware implementing the actual data |
| 99 |
logistics, and serves as a foundation for other future user front-ends, such |
| 100 |
as a web-interface. |
| 101 |
|
| 102 |
.. |---| unicode:: U+02014 .. em dash |
| 103 |
|
| 104 |
.. _Git: https://git-scm.com |
| 105 |
.. _git-annex: http://git-annex.branchable.com |