Debian Codegraph @

docs/source/basics.rst

raw
1 .. -*- mode: rst -*-
2 .. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79:
3
4 .. _chap_basic_principles:
5
6 ****************
7 Basic principles
8 ****************
9
10 DataLad is designed to be used both as a command-line tool, and as a Python
11 module. The sections :ref:`chap_cmdline` and :ref:`chap_modref` provide
12 detailed description of the commands and functions of the two interfaces. This
13 section presents common concepts. Although examples will frequently be
14 presented using command line interface commands, all functionality with
15 identically named functions and options are available through Python API as
16 well.
17
18 Datasets
19 ========
20
21 A DataLad :term:`dataset` is a Git repository that may or may not have a data
22 :term:`annex` that is used to manage data referenced in a dataset. In practice,
23 most DataLad datasets will come with an annex.
24
25 Types of IDs used in datasets
26 -----------------------------
27
28 Four types of unique identifiers are used by DataLad to enable identification
29 of different aspects of datasets and their components.
30
31 Dataset ID
32 A UUID that identifies a dataset as a whole across its entire history and
33 flavors. This ID is stored in a dataset's own configuration file
34 (``<dataset root>/.datalad/config``) under the configuration key
35 ``datalad.dataset.id``.
36 As this configuration is stored in a file that is part of the Git history of
37 a dataset, this ID is identical for all "clones" of a dataset and across all
38 its versions. If the purpose or scope of a dataset changes enough to warrant
39 a new dataset ID, it can be changed by altering the dataset configuration
40 setting.
41 Annex ID
42 A UUID assigned to an annex of each individual clone of a dataset repository.
43 Git-annex uses this UUID to track file content availability information. The
44 UUID is available under the configuration key ``annex.uuid`` and is stored
45 in the configuration file of a local clone (``<dataset root>/.git/config``).
46 A single dataset instance (i.e. clone) can only have a single annex UUID,
47 but a dataset with multiple clones will have multiple annex UUIDs.
48 Commit ID
49 A Git hexsha or tag that identifies a version of a dataset. This ID uniquely
50 identifies the content and history of a dataset up to its present state. As
51 the dataset history also includes the dataset ID, a commit ID of a DataLad
52 dataset is unique to a particular dataset.
53 Content ID
54 Git-annex key (typically a checksum) assigned to the content of a file in
55 a dataset's annex. The checksum reflects the content of a file, not its name.
56 Hence the content of multiple identical files in a single (or across)
57 dataset(s) will have the same checksum. Content IDs are managed by Git-annex
58 in a dedicated ``annex`` branch of the dataset's Git repository.
59
60
61 Dataset nesting
62 ---------------
63
64 Datasets can contain other datasets (:term:`subdataset`\s), which can in turn
65 contain subdatasets, and so on. There is no limit to the depth of nesting
66 datasets. Each dataset in such a hierarchy has its own annex and its own
67 history. The parent or :term:`superdataset` only tracks the specific state of a
68 subdataset, and information on where it can be obtained. This is a powerful yet
69 lightweight mechanism for combining multiple individual datasets for a specific
70 purpose, such as the combination of source code repositories with other
71 resources for a tailored application. In many cases DataLad can work with a
72 hierarchy of datasets just as if it were a single dataset. Here is a demo:
73
74 .. include:: basics_nesteddatasets.rst.in
75 :start-after: Let's create a dataset
76 :end-before: ___________________________
77
78
79 Dataset collections
80 -------------------
81
82 A superdataset can also be seen as a curated collection of datasets, for example,
83 for a certain data modality, a field of science, a certain author, or from
84 one project (maybe the resource for a movie production). This lightweight
85 coupling between super and subdatasets enables scenarios where individual datasets
86 are maintained by a disjoint set of people, and the dataset collection itself can
87 be curated by a completely independent entity. Any individual dataset can be
88 part of any number of such collections.
89
90 Benefiting from Git's support for workflows based on decentralized "clones" of
91 a repository, DataLad's datasets can be (re-)published to a new location
92 without losing the connection between the "original" and the new "copy". This
93 is extremely useful for collaborative work, but also in more mundane scenarios
94 such as data backup, or temporary deployment of a dataset on a compute cluster,
95 or in the cloud. Using git-annex, data can also get synchronized across
96 different locations of a dataset (:term:`sibling`\s in DataLad terminology).
97 Using metadata tags, it is even possible to configure different levels of
98 desired data redundancy across the network of dataset, or to prevent
99 publication of sensitive data to publicly accessible repositories. Individual
100 datasets in a hierarchy of (sub)datasets need not be stored at the same location.
101 Continuing with an earlier example, it is possible to post a curated
102 collection of datasets, as a superdataset, on GitHub, while the actual datasets
103 live on different servers all around the world.
104
105 Basic command line usage
106 ========================
107
108 .. include:: basics_cmdline.rst.in
109 :end-before: ___________________________
110
111
112 API principles
113 ==============
114
115 You can use DataLad's ``install`` command to download datasets. The command accepts
116 URLs of different protocols (``http``, ``ssh``) as an argument. Nevertheless, the easiest way
117 to obtain a first dataset is downloading the default :term:`superdataset` from
118 https://datasets.datalad.org/ using a shortcut.
119
120 Downloading DataLad's default superdataset
121 --------------------------------------------
122
123 https://datasets.datalad.org provides a super-dataset consisting of datasets
124 from various portals and sites. Many of them were crawled, and periodically
125 updated, using `datalad-crawler <https://github.com/datalad/datalad-crawler>`__
126 extension. The argument ``///`` can be used
127 as a shortcut that points to the superdataset located at https://datasets.datalad.org/.
128 Here are three common examples in command line notation:
129
130 ``datalad install ///``
131 installs this superdataset (metadata without subdatasets) in a
132 `datasets.datalad.org/` subdirectory under the current directory
133 ``datalad install -r ///openfmri``
134 installs the openfmri superdataset into an `openfmri/` subdirectory.
135 Additionally, the ``-r`` flag recursively downloads all metadata of datasets
136 available from http://openfmri.org as subdatasets into the `openfmri/` subdirectory
137 ``datalad install -g -J3 -r ///labs/haxby``
138 installs the superdataset of datasets released by the lab of Dr. James V. Haxby
139 and all subdatasets' metadata. The ``-g`` flag indicates getting the actual data, too.
140 It does so by using 3 parallel download processes (``-J3`` flag).
141
142
143 Downloading datasets via http
144 -----------------------------
145
146 In most places where DataLad accepts URLs as arguments these URLs can be
147 regular ``http`` or ``https`` protocol URLs. For example:
148
149 ``datalad install https://github.com/psychoinformatics-de/studyforrest-data-phase2.git``
150
151 Downloading datasets via ssh
152 ----------------------------
153 DataLad also supports SSH URLs, such as ``ssh://me@localhost/path``.
154
155 ``datalad install ssh://me@localhost/path``
156
157 Finally, DataLad supports SSH login style resource identifiers, such as ``me@localhost:/path``.
158
159 ``datalad install me@localhost:/path``
160
161
162 Commands `install` vs `get`
163 ---------------------------
164
165 The ``install`` and ``get`` commands might seem confusingly similar at first.
166 Both of them could be used to install any number of subdatasets, and fetch
167 content of the data files. Differences lie primarily in their default
168 behaviour and outputs, and thus intended use. Both ``install`` and ``get``
169 take local paths as their arguments, but their default behavior and output
170 might differ;
171
172 - **install** primarily operates and reports at the level of **datasets**, and
173 returns as a result dataset(s)
174 which either were just installed, or were installed previously already under
175 specified locations. So result should be the same if the same ``install``
176 command ran twice on the same datasets. It **does not fetch** data files by
177 default
178
179 - **get** primarily operates at the level of **paths** (datasets, directories, and/or
180 files). As a result it returns only what was installed (datasets) or fetched
181 (files). So result of rerunning the same ``get`` command should report that
182 nothing new was installed or fetched. It **fetches** data files by default.
183
184 In how both commands operate on provided paths, it could be said that ``install
185 == get -n``, and ``install -g == get``. But ``install`` also has ability to
186 install new datasets from remote locations given their URLs (e.g.,
187 ``https://datasets.datalad.org/`` for our super-dataset) and SSH targets (e.g.,
188 ``[login@]host:path``) if they are provided as the argument to its call or
189 explicitly as ``--source`` option. If ``datalad install --source URL
190 DESTINATION`` (command line example) is used, then dataset from URL gets
191 installed under PATH. In case of ``datalad install URL`` invocation, PATH is
192 taken from the last name within URL similar to how ``git clone`` does it. If
193 former specification allows to specify only a single URL and a PATH at a time,
194 later one can take multiple remote locations from which datasets could be
195 installed.
196
197 So, as a rule of thumb -- if you want to install from external URL or fetch a
198 sub-dataset without downloading data files stored under annex -- use ``install``.
199 In Python API ``install`` is also to be used when you want to receive in output the
200 corresponding Dataset object to operate on, and be able to use it even if you
201 rerun the script. In all other cases, use ``get``.