| 1 |
.. -*- mode: rst -*- |
| 2 |
.. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79: |
| 3 |
|
| 4 |
.. _chap_basic_principles: |
| 5 |
|
| 6 |
**************** |
| 7 |
Basic principles |
| 8 |
**************** |
| 9 |
|
| 10 |
DataLad is designed to be used both as a command-line tool, and as a Python |
| 11 |
module. The sections :ref:`chap_cmdline` and :ref:`chap_modref` provide |
| 12 |
detailed description of the commands and functions of the two interfaces. This |
| 13 |
section presents common concepts. Although examples will frequently be |
| 14 |
presented using command line interface commands, all functionality with |
| 15 |
identically named functions and options are available through Python API as |
| 16 |
well. |
| 17 |
|
| 18 |
Datasets |
| 19 |
======== |
| 20 |
|
| 21 |
A DataLad :term:`dataset` is a Git repository that may or may not have a data |
| 22 |
:term:`annex` that is used to manage data referenced in a dataset. In practice, |
| 23 |
most DataLad datasets will come with an annex. |
| 24 |
|
| 25 |
Types of IDs used in datasets |
| 26 |
----------------------------- |
| 27 |
|
| 28 |
Four types of unique identifiers are used by DataLad to enable identification |
| 29 |
of different aspects of datasets and their components. |
| 30 |
|
| 31 |
Dataset ID |
| 32 |
A UUID that identifies a dataset as a whole across its entire history and |
| 33 |
flavors. This ID is stored in a dataset's own configuration file |
| 34 |
(``<dataset root>/.datalad/config``) under the configuration key |
| 35 |
``datalad.dataset.id``. |
| 36 |
As this configuration is stored in a file that is part of the Git history of |
| 37 |
a dataset, this ID is identical for all "clones" of a dataset and across all |
| 38 |
its versions. If the purpose or scope of a dataset changes enough to warrant |
| 39 |
a new dataset ID, it can be changed by altering the dataset configuration |
| 40 |
setting. |
| 41 |
Annex ID |
| 42 |
A UUID assigned to an annex of each individual clone of a dataset repository. |
| 43 |
Git-annex uses this UUID to track file content availability information. The |
| 44 |
UUID is available under the configuration key ``annex.uuid`` and is stored |
| 45 |
in the configuration file of a local clone (``<dataset root>/.git/config``). |
| 46 |
A single dataset instance (i.e. clone) can only have a single annex UUID, |
| 47 |
but a dataset with multiple clones will have multiple annex UUIDs. |
| 48 |
Commit ID |
| 49 |
A Git hexsha or tag that identifies a version of a dataset. This ID uniquely |
| 50 |
identifies the content and history of a dataset up to its present state. As |
| 51 |
the dataset history also includes the dataset ID, a commit ID of a DataLad |
| 52 |
dataset is unique to a particular dataset. |
| 53 |
Content ID |
| 54 |
Git-annex key (typically a checksum) assigned to the content of a file in |
| 55 |
a dataset's annex. The checksum reflects the content of a file, not its name. |
| 56 |
Hence the content of multiple identical files in a single (or across) |
| 57 |
dataset(s) will have the same checksum. Content IDs are managed by Git-annex |
| 58 |
in a dedicated ``annex`` branch of the dataset's Git repository. |
| 59 |
|
| 60 |
|
| 61 |
Dataset nesting |
| 62 |
--------------- |
| 63 |
|
| 64 |
Datasets can contain other datasets (:term:`subdataset`\s), which can in turn |
| 65 |
contain subdatasets, and so on. There is no limit to the depth of nesting |
| 66 |
datasets. Each dataset in such a hierarchy has its own annex and its own |
| 67 |
history. The parent or :term:`superdataset` only tracks the specific state of a |
| 68 |
subdataset, and information on where it can be obtained. This is a powerful yet |
| 69 |
lightweight mechanism for combining multiple individual datasets for a specific |
| 70 |
purpose, such as the combination of source code repositories with other |
| 71 |
resources for a tailored application. In many cases DataLad can work with a |
| 72 |
hierarchy of datasets just as if it were a single dataset. Here is a demo: |
| 73 |
|
| 74 |
.. include:: basics_nesteddatasets.rst.in |
| 75 |
:start-after: Let's create a dataset |
| 76 |
:end-before: ___________________________ |
| 77 |
|
| 78 |
|
| 79 |
Dataset collections |
| 80 |
------------------- |
| 81 |
|
| 82 |
A superdataset can also be seen as a curated collection of datasets, for example, |
| 83 |
for a certain data modality, a field of science, a certain author, or from |
| 84 |
one project (maybe the resource for a movie production). This lightweight |
| 85 |
coupling between super and subdatasets enables scenarios where individual datasets |
| 86 |
are maintained by a disjoint set of people, and the dataset collection itself can |
| 87 |
be curated by a completely independent entity. Any individual dataset can be |
| 88 |
part of any number of such collections. |
| 89 |
|
| 90 |
Benefiting from Git's support for workflows based on decentralized "clones" of |
| 91 |
a repository, DataLad's datasets can be (re-)published to a new location |
| 92 |
without losing the connection between the "original" and the new "copy". This |
| 93 |
is extremely useful for collaborative work, but also in more mundane scenarios |
| 94 |
such as data backup, or temporary deployment of a dataset on a compute cluster, |
| 95 |
or in the cloud. Using git-annex, data can also get synchronized across |
| 96 |
different locations of a dataset (:term:`sibling`\s in DataLad terminology). |
| 97 |
Using metadata tags, it is even possible to configure different levels of |
| 98 |
desired data redundancy across the network of dataset, or to prevent |
| 99 |
publication of sensitive data to publicly accessible repositories. Individual |
| 100 |
datasets in a hierarchy of (sub)datasets need not be stored at the same location. |
| 101 |
Continuing with an earlier example, it is possible to post a curated |
| 102 |
collection of datasets, as a superdataset, on GitHub, while the actual datasets |
| 103 |
live on different servers all around the world. |
| 104 |
|
| 105 |
Basic command line usage |
| 106 |
======================== |
| 107 |
|
| 108 |
.. include:: basics_cmdline.rst.in |
| 109 |
:end-before: ___________________________ |
| 110 |
|
| 111 |
|
| 112 |
API principles |
| 113 |
============== |
| 114 |
|
| 115 |
You can use DataLad's ``install`` command to download datasets. The command accepts |
| 116 |
URLs of different protocols (``http``, ``ssh``) as an argument. Nevertheless, the easiest way |
| 117 |
to obtain a first dataset is downloading the default :term:`superdataset` from |
| 118 |
https://datasets.datalad.org/ using a shortcut. |
| 119 |
|
| 120 |
Downloading DataLad's default superdataset |
| 121 |
-------------------------------------------- |
| 122 |
|
| 123 |
https://datasets.datalad.org provides a super-dataset consisting of datasets |
| 124 |
from various portals and sites. Many of them were crawled, and periodically |
| 125 |
updated, using `datalad-crawler <https://github.com/datalad/datalad-crawler>`__ |
| 126 |
extension. The argument ``///`` can be used |
| 127 |
as a shortcut that points to the superdataset located at https://datasets.datalad.org/. |
| 128 |
Here are three common examples in command line notation: |
| 129 |
|
| 130 |
``datalad install ///`` |
| 131 |
installs this superdataset (metadata without subdatasets) in a |
| 132 |
`datasets.datalad.org/` subdirectory under the current directory |
| 133 |
``datalad install -r ///openfmri`` |
| 134 |
installs the openfmri superdataset into an `openfmri/` subdirectory. |
| 135 |
Additionally, the ``-r`` flag recursively downloads all metadata of datasets |
| 136 |
available from http://openfmri.org as subdatasets into the `openfmri/` subdirectory |
| 137 |
``datalad install -g -J3 -r ///labs/haxby`` |
| 138 |
installs the superdataset of datasets released by the lab of Dr. James V. Haxby |
| 139 |
and all subdatasets' metadata. The ``-g`` flag indicates getting the actual data, too. |
| 140 |
It does so by using 3 parallel download processes (``-J3`` flag). |
| 141 |
|
| 142 |
|
| 143 |
Downloading datasets via http |
| 144 |
----------------------------- |
| 145 |
|
| 146 |
In most places where DataLad accepts URLs as arguments these URLs can be |
| 147 |
regular ``http`` or ``https`` protocol URLs. For example: |
| 148 |
|
| 149 |
``datalad install https://github.com/psychoinformatics-de/studyforrest-data-phase2.git`` |
| 150 |
|
| 151 |
Downloading datasets via ssh |
| 152 |
---------------------------- |
| 153 |
DataLad also supports SSH URLs, such as ``ssh://me@localhost/path``. |
| 154 |
|
| 155 |
``datalad install ssh://me@localhost/path`` |
| 156 |
|
| 157 |
Finally, DataLad supports SSH login style resource identifiers, such as ``me@localhost:/path``. |
| 158 |
|
| 159 |
``datalad install me@localhost:/path`` |
| 160 |
|
| 161 |
|
| 162 |
Commands `install` vs `get` |
| 163 |
--------------------------- |
| 164 |
|
| 165 |
The ``install`` and ``get`` commands might seem confusingly similar at first. |
| 166 |
Both of them could be used to install any number of subdatasets, and fetch |
| 167 |
content of the data files. Differences lie primarily in their default |
| 168 |
behaviour and outputs, and thus intended use. Both ``install`` and ``get`` |
| 169 |
take local paths as their arguments, but their default behavior and output |
| 170 |
might differ; |
| 171 |
|
| 172 |
- **install** primarily operates and reports at the level of **datasets**, and |
| 173 |
returns as a result dataset(s) |
| 174 |
which either were just installed, or were installed previously already under |
| 175 |
specified locations. So result should be the same if the same ``install`` |
| 176 |
command ran twice on the same datasets. It **does not fetch** data files by |
| 177 |
default |
| 178 |
|
| 179 |
- **get** primarily operates at the level of **paths** (datasets, directories, and/or |
| 180 |
files). As a result it returns only what was installed (datasets) or fetched |
| 181 |
(files). So result of rerunning the same ``get`` command should report that |
| 182 |
nothing new was installed or fetched. It **fetches** data files by default. |
| 183 |
|
| 184 |
In how both commands operate on provided paths, it could be said that ``install |
| 185 |
== get -n``, and ``install -g == get``. But ``install`` also has ability to |
| 186 |
install new datasets from remote locations given their URLs (e.g., |
| 187 |
``https://datasets.datalad.org/`` for our super-dataset) and SSH targets (e.g., |
| 188 |
``[login@]host:path``) if they are provided as the argument to its call or |
| 189 |
explicitly as ``--source`` option. If ``datalad install --source URL |
| 190 |
DESTINATION`` (command line example) is used, then dataset from URL gets |
| 191 |
installed under PATH. In case of ``datalad install URL`` invocation, PATH is |
| 192 |
taken from the last name within URL similar to how ``git clone`` does it. If |
| 193 |
former specification allows to specify only a single URL and a PATH at a time, |
| 194 |
later one can take multiple remote locations from which datasets could be |
| 195 |
installed. |
| 196 |
|
| 197 |
So, as a rule of thumb -- if you want to install from external URL or fetch a |
| 198 |
sub-dataset without downloading data files stored under annex -- use ``install``. |
| 199 |
In Python API ``install`` is also to be used when you want to receive in output the |
| 200 |
corresponding Dataset object to operate on, and be able to use it even if you |
| 201 |
rerun the script. In all other cases, use ``get``. |