Debian Codegraph @

docs/source/related.rst

raw
1 Delineation from related solutions
2 **********************************
3
4 To our knowledge, there is no other effort with a scope as broad as DataLad's.
5 DataLad aims to unify access to vast arrays of (scientific) data in a domain and
6 data modality agnostic fashion with as few and universally available software
7 dependencies as possible.
8
9 The most comparable project regarding the idea of federating access to various
10 data providers is the iRODS_-based `INCF Dataspace`_ project. IRODS is a
11 powerful, NSF-supported framework, but it requires non-trivial deployment and
12 management procedures. As a representative of *data grid* technology, it is
13 more suitable for an institutional deployment, as data access, authentication,
14 permission management, and versioning are complex and not-feasible to be
15 performed directly by researchers. DataLad on the other hand federates
16 institutionally hosted data, but in addition enables individual researchers and
17 small labs to contribute datasets to the federation with minimal cost and
18 without the need for centralized coordination and permission management.
19
20 .. _IRODS: https://irods.org
21 .. _INCF Dataspace: http://www.incf.org/resources/data-space
22
23
24 Data catalogs
25 =============
26
27 Existing data-portals, such as DataDryad_, or domain-specific ones (e.g. `Human
28 Connectome`_, OpenfMRI_), concentrate on collecting, cataloging, and making
29 data available. They offer an abstraction from local data management
30 peculiarities (organization, updates, sharing). Ad-hoc collections of pointers
31 to available data, such as `reddit datasets`_ and `Inside-R datasets`_, do not
32 provide any unified interface to assemble and manage such data. Data portals
33 can be used as seed information and data providers for DataLad. These portals
34 could in turn adopt DataLad to expose readily usable data collections via a
35 federated infrastructure.
36
37 .. _Human Connectome: http://www.humanconnectomeproject.org
38 .. _OpenfMRI: http://openfmri.org
39 .. _DataDryad: http://datadryad.org
40 .. _reddit datasets: http://www.reddit.com/r/datasets
41 .. _Inside-R datasets: http://www.inside-r.org/howto/finding-data-internet
42
43
44 Data delivery/management middleware
45 ===================================
46
47 Even though there are projects to manage data directly with dVCS (e.g. Git),
48 such as the `Rdatasets Git repository`_ this approach does not scale, for example
49 to the amount of data typically observed in a scientific context. DataLad
50 uses git-annex_ to support managing large amounts of data with Git, while
51 avoiding the scalability issues of putting data directly into Git repositories.
52
53 In scientific software development, frequently using Git for source code
54 management, many projects are also confronted with the problem of managing
55 large data arrays needed, for example, for software testing. An exemplar
56 project is `ITK Data`_ which is conceptually similar to git-annex: data content
57 is referenced by unique keys (checksums), which are made redundantly available
58 through multiple remote key-store farms and can be obtained using specialized
59 functionality in the CMake software build system. However, the scope of this
60 project is limited to software QA, and only provides an ad-hoc collection of
61 guidelines and supporting scripts.
62
63 .. _Rdatasets Git repository: http://github.com/vincentarelbundock/Rdatasets
64 .. _ITK Data: http://www.itk.org/Wiki/ITK/Git/Develop/Data
65
66 The git-annex website provides a comparison_ of Git-annex to other available
67 distributed data management tools, such as git-media_, git-fat_, and others.
68 None of the alternative frameworks provides all of the features of git-annex,
69 such as integration with native Git workflows, distributed redundant storage,
70 and partial checkouts in one project. Additional features of git-annex which
71 are not necessarily needed by DataLad (git-annex assistant, encryption support,
72 etc.) make it even more appealing for extended coverage of numerous scenarios.
73 Moreover, neither of the alternative solutions has already reached a maturity,
74 availability, and level of adoption that would be comparable to that of
75 git-annex.
76
77 .. _git-annex: http://git-annex.branchable.com
78 .. _comparison: http://git-annex.branchable.com/not
79 .. _git-media: https://github.com/schacon/git-media
80 .. _git-fat: https://github.com/jedbrown/git-fat
81
82 .. _chap-git-annex-datalad-comparison:
83
84 Git/Git-annex/DataLad
85 =====================
86
87 Although it is possible, and intended, to use DataLad without ever invoking git
88 or git-annex commands directly, it is useful to appreciate that DataLad is
89 build atop of very flexible and powerful tools. Knowing basics of git and
90 git-annex in addition to DataLad helps to not only make better use of
91 DataLad but also to enable more advanced and more efficient data management
92 scenarios. DataLad makes use of lower-level configuration and data structures
93 as much as possible. Consequently, it is possible to manipulate DataLad
94 datasets with low-level tools if needed. Moreover, DataLad datasets are
95 compatible with tools and services designed to work with plain Git repositories,
96 such as the popular GitHub_ service.
97
98 .. _github: https://github.com
99
100 To better illustrate the different scopes, the following table provides an
101 overview of the features that are contributed by each software technology
102 layer.
103
104 ================================================ ============= =============== ==============
105 Feature Git Git-annex DataLad
106 ================================================ ============= =============== ==============
107 Version control (text, code) |tup| |tup| can mix |tup| can mix
108 Version control (binary data) (not advised) |tup| |tup|
109 Auto-crawling available resources |tup| RSS feeds |tup| flexible
110 Unified dataset handling |tup|
111 - recursive operation on datasets |tup|
112 - seamless operation across datasets boundaries |tup|
113 - meta-data support |tup| per-file |tup|
114 - meta-data aggregation |tup| flexible
115 Unified authentication interface |tup|
116 ================================================ ============= =============== ==============
117
118 .. |tup| unicode:: U+2713 .. check mark
119 :trim: