Debian Codegraph @

docs/source/background.rst

raw
1 Background and motivation
2 *************************
3
4 Vision
5 ======
6
7 Data is at the core of science, and unobstructed access promotes scientific
8 discovery through collaboration between data producers and consumers. The last
9 years have seen dramatic improvements in availability of data resources for
10 collaborative research, and new data providers are becoming available all the
11 time.
12
13 However, despite the increased availability of data, their accessibility is far
14 from being optimal. Potential consumers of these public datasets have to
15 manually browse various disconnected warehouses with heterogeneous interfaces.
16 Once obtained, data is disconnected from its origin and data versioning is
17 often ad-hoc or completely absent. If data consumers can be reliably informed
18 about data updates at all, review of changes is difficult, and re-deployment is
19 tedious and error-prone. This leads to wasteful friction caused by outdated or
20 faulty data.
21
22 The vision for this project is to transform the state of data-sharing and
23 collaborative work by providing uniform access to available datasets --
24 independent of hosting solutions or authentication schemes -- with reliable
25 versioning and versatile deployment logistics. This is achieved by means of a
26 :term:`dataset` handle, a lightweight representation of a dataset
27 that is capable of tracking the identity and location of a dataset's content as
28 well as carry meta-data. Together with associated software tools, scientists
29 are able to obtain, use, extend, and share datasets (or parts thereof) in a
30 way that is traceable back to the original data producer and is therefore
31 capable of establishing a strong connection between data consumers and the
32 evolution of a dataset by future extension or error correction.
33
34 Moreover, DataLad aims to provide all tools necessary to create and publish
35 *data distributions* |---| an analog to software distributions or app-stores
36 that provide logistics middleware for software deployment. Scientific
37 communities can use these tools to gather, curate, and make publicly available
38 specialized collections of datasets for specific research topics or data
39 modalities. All of this is possible by leveraging existing data sharing
40 platforms and institutional resources without the need for funding extra
41 infrastructure of duplicate storage. Specifically, this project aims to provide
42 a comprehensive, extensible data distribution for neuroscientific datasets that
43 is kept up-to-date by an automated service.
44
45
46 Technological foundation: git-annex
47 ===================================
48
49 The outlined task is not unique to the problem of data-sharing in science.
50 Logistical challenges such as delivering data, long-term storage and archiving,
51 identity tracking, and synchronization between multiple sites are rather
52 common. Consequently, solutions have been developed in other contexts that can
53 be adapted to benefit scientific data-sharing.
54
55 The closest match is the software tool git-annex_. It combines the features of
56 the distributed version control system (dVCS) Git_ |---| a technology that has
57 revolutionized collaborative software development -- with versatile data access
58 and delivery logistics. Git-annex was originally developed to address use cases
59 such as managing a collection of family pictures at home. With git-annex, any
60 family member can obtain an individual copy of such a picture library |---| the
61 :term:`annex`. The annex in this example is essentially an image repository
62 that presents individual pictures to users as files in a single directory
63 structure, even though the actual image file contents may be distributed across
64 multiple locations, including a home-server, cloud-storage, or even off-line
65 media such as external hard-drives.
66
67 Git-annex provides functionality to obtain file contents upon request and can
68 prompt users to make particular storage devices available when needed (e.g. a
69 backup hard-drive kept in a fire-proof compartment). Git-annex can also remove
70 files from a local copy of that image repository, for example to free up space
71 on a laptop, while ensuring a configurable level of data redundancy across all
72 known storage locations. Lastly, git-annex is able to synchronize the content
73 of multiple distributed copies of this image repository, for example in order
74 to incorporate images added with the git-annex on the laptop of another family
75 member. It is important to note that git-annex is agnostic of the actual file
76 types and is not limited to images.
77
78 We believe that the approach to data logistics taken by git-annex and the
79 functionality it is currently providing are an ideal middleware for scientific
80 data-sharing. Its data repository model :term:`annex` readily provides the
81 majority of principal features needed for a dataset handle such as history
82 recording, identity tracking, and item-based resource locators. Consequently,
83 instead of a from-scratch development, required features, such as dedicated
84 support for existing data-sharing portals and dataset meta-information, can be
85 added to a working solution that is already in production for several years.
86 As a result, DataLad focuses on the expansion of git-annex's functionality and
87 the development of tools that build atop Git and git-annex and enable the
88 creation, management, use, and publication of dataset handles and collections
89 thereof.
90
91 Objective
92 =========
93
94 Building atop git-annex, DataLad aims to provide a single, uniform interface to
95 access data from various data-sharing initiatives and data providers, and
96 functionality to create, deliver, update, and share datasets for individuals
97 and portal maintainers. As a command-line tool, it provides an abstraction
98 layer for the underlying Git-based middleware implementing the actual data
99 logistics, and serves as a foundation for other future user front-ends, such
100 as a web-interface.
101
102 .. |---| unicode:: U+02014 .. em dash
103
104 .. _Git: https://git-scm.com
105 .. _git-annex: http://git-annex.branchable.com