| 1 |
Delineation from related solutions |
| 2 |
********************************** |
| 3 |
|
| 4 |
To our knowledge, there is no other effort with a scope as broad as DataLad's. |
| 5 |
DataLad aims to unify access to vast arrays of (scientific) data in a domain and |
| 6 |
data modality agnostic fashion with as few and universally available software |
| 7 |
dependencies as possible. |
| 8 |
|
| 9 |
The most comparable project regarding the idea of federating access to various |
| 10 |
data providers is the iRODS_-based `INCF Dataspace`_ project. IRODS is a |
| 11 |
powerful, NSF-supported framework, but it requires non-trivial deployment and |
| 12 |
management procedures. As a representative of *data grid* technology, it is |
| 13 |
more suitable for an institutional deployment, as data access, authentication, |
| 14 |
permission management, and versioning are complex and not-feasible to be |
| 15 |
performed directly by researchers. DataLad on the other hand federates |
| 16 |
institutionally hosted data, but in addition enables individual researchers and |
| 17 |
small labs to contribute datasets to the federation with minimal cost and |
| 18 |
without the need for centralized coordination and permission management. |
| 19 |
|
| 20 |
.. _IRODS: https://irods.org |
| 21 |
.. _INCF Dataspace: http://www.incf.org/resources/data-space |
| 22 |
|
| 23 |
|
| 24 |
Data catalogs |
| 25 |
============= |
| 26 |
|
| 27 |
Existing data-portals, such as DataDryad_, or domain-specific ones (e.g. `Human |
| 28 |
Connectome`_, OpenfMRI_), concentrate on collecting, cataloging, and making |
| 29 |
data available. They offer an abstraction from local data management |
| 30 |
peculiarities (organization, updates, sharing). Ad-hoc collections of pointers |
| 31 |
to available data, such as `reddit datasets`_ and `Inside-R datasets`_, do not |
| 32 |
provide any unified interface to assemble and manage such data. Data portals |
| 33 |
can be used as seed information and data providers for DataLad. These portals |
| 34 |
could in turn adopt DataLad to expose readily usable data collections via a |
| 35 |
federated infrastructure. |
| 36 |
|
| 37 |
.. _Human Connectome: http://www.humanconnectomeproject.org |
| 38 |
.. _OpenfMRI: http://openfmri.org |
| 39 |
.. _DataDryad: http://datadryad.org |
| 40 |
.. _reddit datasets: http://www.reddit.com/r/datasets |
| 41 |
.. _Inside-R datasets: http://www.inside-r.org/howto/finding-data-internet |
| 42 |
|
| 43 |
|
| 44 |
Data delivery/management middleware |
| 45 |
=================================== |
| 46 |
|
| 47 |
Even though there are projects to manage data directly with dVCS (e.g. Git), |
| 48 |
such as the `Rdatasets Git repository`_ this approach does not scale, for example |
| 49 |
to the amount of data typically observed in a scientific context. DataLad |
| 50 |
uses git-annex_ to support managing large amounts of data with Git, while |
| 51 |
avoiding the scalability issues of putting data directly into Git repositories. |
| 52 |
|
| 53 |
In scientific software development, frequently using Git for source code |
| 54 |
management, many projects are also confronted with the problem of managing |
| 55 |
large data arrays needed, for example, for software testing. An exemplar |
| 56 |
project is `ITK Data`_ which is conceptually similar to git-annex: data content |
| 57 |
is referenced by unique keys (checksums), which are made redundantly available |
| 58 |
through multiple remote key-store farms and can be obtained using specialized |
| 59 |
functionality in the CMake software build system. However, the scope of this |
| 60 |
project is limited to software QA, and only provides an ad-hoc collection of |
| 61 |
guidelines and supporting scripts. |
| 62 |
|
| 63 |
.. _Rdatasets Git repository: http://github.com/vincentarelbundock/Rdatasets |
| 64 |
.. _ITK Data: http://www.itk.org/Wiki/ITK/Git/Develop/Data |
| 65 |
|
| 66 |
The git-annex website provides a comparison_ of Git-annex to other available |
| 67 |
distributed data management tools, such as git-media_, git-fat_, and others. |
| 68 |
None of the alternative frameworks provides all of the features of git-annex, |
| 69 |
such as integration with native Git workflows, distributed redundant storage, |
| 70 |
and partial checkouts in one project. Additional features of git-annex which |
| 71 |
are not necessarily needed by DataLad (git-annex assistant, encryption support, |
| 72 |
etc.) make it even more appealing for extended coverage of numerous scenarios. |
| 73 |
Moreover, neither of the alternative solutions has already reached a maturity, |
| 74 |
availability, and level of adoption that would be comparable to that of |
| 75 |
git-annex. |
| 76 |
|
| 77 |
.. _git-annex: http://git-annex.branchable.com |
| 78 |
.. _comparison: http://git-annex.branchable.com/not |
| 79 |
.. _git-media: https://github.com/schacon/git-media |
| 80 |
.. _git-fat: https://github.com/jedbrown/git-fat |
| 81 |
|
| 82 |
.. _chap-git-annex-datalad-comparison: |
| 83 |
|
| 84 |
Git/Git-annex/DataLad |
| 85 |
===================== |
| 86 |
|
| 87 |
Although it is possible, and intended, to use DataLad without ever invoking git |
| 88 |
or git-annex commands directly, it is useful to appreciate that DataLad is |
| 89 |
build atop of very flexible and powerful tools. Knowing basics of git and |
| 90 |
git-annex in addition to DataLad helps to not only make better use of |
| 91 |
DataLad but also to enable more advanced and more efficient data management |
| 92 |
scenarios. DataLad makes use of lower-level configuration and data structures |
| 93 |
as much as possible. Consequently, it is possible to manipulate DataLad |
| 94 |
datasets with low-level tools if needed. Moreover, DataLad datasets are |
| 95 |
compatible with tools and services designed to work with plain Git repositories, |
| 96 |
such as the popular GitHub_ service. |
| 97 |
|
| 98 |
.. _github: https://github.com |
| 99 |
|
| 100 |
To better illustrate the different scopes, the following table provides an |
| 101 |
overview of the features that are contributed by each software technology |
| 102 |
layer. |
| 103 |
|
| 104 |
================================================ ============= =============== ============== |
| 105 |
Feature Git Git-annex DataLad |
| 106 |
================================================ ============= =============== ============== |
| 107 |
Version control (text, code) |tup| |tup| can mix |tup| can mix |
| 108 |
Version control (binary data) (not advised) |tup| |tup| |
| 109 |
Auto-crawling available resources |tup| RSS feeds |tup| flexible |
| 110 |
Unified dataset handling |tup| |
| 111 |
- recursive operation on datasets |tup| |
| 112 |
- seamless operation across datasets boundaries |tup| |
| 113 |
- meta-data support |tup| per-file |tup| |
| 114 |
- meta-data aggregation |tup| flexible |
| 115 |
Unified authentication interface |tup| |
| 116 |
================================================ ============= =============== ============== |
| 117 |
|
| 118 |
.. |tup| unicode:: U+2713 .. check mark |
| 119 |
:trim: |