Debian Codegraph @

docs/source/design/provenance_capture.rst

raw
1 .. -*- mode: rst -*-
2 .. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79:
3
4 .. _chap_design_provenance_capture:
5
6 ******************
7 Provenance capture
8 ******************
9
10 .. topic:: Specification scope and status
11
12 This specification describes the current implementation.
13
14 The ability to capture process provenance---the information what activity
15 initiated by which entity yielded which outputs, given a set of parameters, a
16 computational environment, and potential input data---is a core feature of
17 DataLad.
18
19 Provenance capture is supported for any computational process that can be
20 expressed as a command line call. The simplest form of provenance tracking can
21 be implemented by prefixing any such a command line call with ``datalad run
22 ...``. When executed in the content of a dataset (with the current working
23 directory typically being in the root of a dataset), DataLad will then:
24
25 1. check the dataset for any unsaved modifications
26 2. execute the given command, when no modifications were found
27 3. save any changes to the dataset that exist after the command has exited without error
28
29 The saved changes are annotated with a structured record that, at minimum,
30 contains the executed command.
31
32 This kind of usage is sufficient for building up an annotated history of a
33 dataset, where all relevant modifications are clearly associated with the
34 commands that caused them. By providing more, optional, information to the
35 ``run`` command, such as a declaration of inputs and outputs, provenance
36 records can be further enriched. This enables additional functionality, such as
37 the automated re-execution of captured processes.
38
39
40 The provenance record
41 =====================
42
43 A DataLad provenance record is a key-value mapping comprising the following
44 main items:
45
46 - ``cmd``: executed command, which may contain placeholders
47 - ``dsid``: DataLad ID of dataset in whose context the command execution took place
48 - ``exit``: numeric exit code of the command
49 - ``inputs``: a list of (relative) file paths for all declared inputs
50 - ``outputs``: a list of (relative) file paths for all declared outputs
51 - ``pwd``: relative path of the working directory for the command execution
52
53 A provenance record is stored in a JSON-serialized form in one of two locations:
54
55 1. In the body of the commit message created when saving caused the dataset modifications
56 2. In a sidecar file underneath ``.datalad/runinfo`` in the root dataset
57
58 Sidecar files have a filename (``record_id``) that is based on checksum of the
59 provenance record content, and are stored as LZMA-compressed binary files.
60 When a sidecar file is used, its ``record_id`` is added to the commit message,
61 instead of the complete record.
62
63
64 Declaration of inputs and outputs
65 =================================
66
67 While not strictly required, it is possible and recommended to declare all
68 paths for process inputs and outputs of a command execution via the respective
69 options of ``run``.
70
71 For all declared inputs, ``run`` will ensure that their file content is present
72 locally at the required version before executing the command.
73
74 For all declared outputs, ``run`` will ensure that the respective locations are
75 writeable.
76
77 It is recommended to declare inputs and outputs both exhaustively and precise,
78 in order to enable the provenance-based automated re-execution of a command. In
79 case of a future re-execution the dataset content may have changed
80 substantially, and a needlessly broad specification of inputs/outputs may lead
81 to undesirable data transfers.
82
83
84 Placeholders in commands and IO specifications
85 ==============================================
86
87 Both command and input/output specification can employ placeholders that will
88 be expanded before command execution. Placeholders use the syntax of the Python
89 ``format()`` specification. A number of standard placeholders are supported
90 (see the ``run`` documentation for a complete list):
91
92 - ``{pwd}`` will be replaced with the full path of the current working directory
93 - ``{dspath}`` will be replaced with the full path of the dataset that run is invoked on
94 - ``{inputs}`` and ``{outputs}`` expand a space-separated list of the declared input and output paths
95
96 Additionally, custom placeholders can be defined as configuration variables
97 under the prefix ``datalad.run.substitutions.``. For example, a configuration
98 setting ``datalad.run.substitutions.myfile=data.txt`` will cause the
99 placeholder ``{myfile}`` to expand to ``data.txt``.
100
101 Selection of individual items for placeholders that expand to multiple values
102 is possible via the standard Python ``format()`` syntax, for example
103 ``{inputs[0]}``.
104
105
106 Result records emitted by ``run``
107 =================================
108
109 When performing a command execution ``run`` will emit results for:
110
111 1. Input preparation (i.e. downloads)
112 2. Output preparation (i.e. unlocks and removals)
113 3. Command execution
114 4. Dataset modification saving (i.e. additions, deletions, modifications)
115
116 By default, ``run`` will stop on the first error. This means that, for example,
117 any failure to download content will prevent command execution. A failing
118 command will prevent saving a potential dataset modification. This behavior can
119 be altered using the standard ``on_failure`` switch of the ``run`` command.
120
121 The emitted result for the command execution contains the provenance record
122 under the ``run_info`` key.
123
124
125 Implementation details
126 ======================
127
128 Most of the described functionality is implemented by the function
129 :func:`datalad.core.local.run.run_command`. It is interfaced by the ``run``
130 command, but also ``rerun``, a utility for automated re-execution based on
131 provenance records, and ``containers-run`` (provided by the ``container``
132 extension package) for command execution in DataLad-tracked containerized
133 environments. This function has a more complex interface, and supports a wider
134 range of use cases than described here.