| 1 |
.. -*- mode: rst -*- |
| 2 |
.. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79: |
| 3 |
|
| 4 |
.. _chap_design_provenance_capture: |
| 5 |
|
| 6 |
****************** |
| 7 |
Provenance capture |
| 8 |
****************** |
| 9 |
|
| 10 |
.. topic:: Specification scope and status |
| 11 |
|
| 12 |
This specification describes the current implementation. |
| 13 |
|
| 14 |
The ability to capture process provenance---the information what activity |
| 15 |
initiated by which entity yielded which outputs, given a set of parameters, a |
| 16 |
computational environment, and potential input data---is a core feature of |
| 17 |
DataLad. |
| 18 |
|
| 19 |
Provenance capture is supported for any computational process that can be |
| 20 |
expressed as a command line call. The simplest form of provenance tracking can |
| 21 |
be implemented by prefixing any such a command line call with ``datalad run |
| 22 |
...``. When executed in the content of a dataset (with the current working |
| 23 |
directory typically being in the root of a dataset), DataLad will then: |
| 24 |
|
| 25 |
1. check the dataset for any unsaved modifications |
| 26 |
2. execute the given command, when no modifications were found |
| 27 |
3. save any changes to the dataset that exist after the command has exited without error |
| 28 |
|
| 29 |
The saved changes are annotated with a structured record that, at minimum, |
| 30 |
contains the executed command. |
| 31 |
|
| 32 |
This kind of usage is sufficient for building up an annotated history of a |
| 33 |
dataset, where all relevant modifications are clearly associated with the |
| 34 |
commands that caused them. By providing more, optional, information to the |
| 35 |
``run`` command, such as a declaration of inputs and outputs, provenance |
| 36 |
records can be further enriched. This enables additional functionality, such as |
| 37 |
the automated re-execution of captured processes. |
| 38 |
|
| 39 |
|
| 40 |
The provenance record |
| 41 |
===================== |
| 42 |
|
| 43 |
A DataLad provenance record is a key-value mapping comprising the following |
| 44 |
main items: |
| 45 |
|
| 46 |
- ``cmd``: executed command, which may contain placeholders |
| 47 |
- ``dsid``: DataLad ID of dataset in whose context the command execution took place |
| 48 |
- ``exit``: numeric exit code of the command |
| 49 |
- ``inputs``: a list of (relative) file paths for all declared inputs |
| 50 |
- ``outputs``: a list of (relative) file paths for all declared outputs |
| 51 |
- ``pwd``: relative path of the working directory for the command execution |
| 52 |
|
| 53 |
A provenance record is stored in a JSON-serialized form in one of two locations: |
| 54 |
|
| 55 |
1. In the body of the commit message created when saving caused the dataset modifications |
| 56 |
2. In a sidecar file underneath ``.datalad/runinfo`` in the root dataset |
| 57 |
|
| 58 |
Sidecar files have a filename (``record_id``) that is based on checksum of the |
| 59 |
provenance record content, and are stored as LZMA-compressed binary files. |
| 60 |
When a sidecar file is used, its ``record_id`` is added to the commit message, |
| 61 |
instead of the complete record. |
| 62 |
|
| 63 |
|
| 64 |
Declaration of inputs and outputs |
| 65 |
================================= |
| 66 |
|
| 67 |
While not strictly required, it is possible and recommended to declare all |
| 68 |
paths for process inputs and outputs of a command execution via the respective |
| 69 |
options of ``run``. |
| 70 |
|
| 71 |
For all declared inputs, ``run`` will ensure that their file content is present |
| 72 |
locally at the required version before executing the command. |
| 73 |
|
| 74 |
For all declared outputs, ``run`` will ensure that the respective locations are |
| 75 |
writeable. |
| 76 |
|
| 77 |
It is recommended to declare inputs and outputs both exhaustively and precise, |
| 78 |
in order to enable the provenance-based automated re-execution of a command. In |
| 79 |
case of a future re-execution the dataset content may have changed |
| 80 |
substantially, and a needlessly broad specification of inputs/outputs may lead |
| 81 |
to undesirable data transfers. |
| 82 |
|
| 83 |
|
| 84 |
Placeholders in commands and IO specifications |
| 85 |
============================================== |
| 86 |
|
| 87 |
Both command and input/output specification can employ placeholders that will |
| 88 |
be expanded before command execution. Placeholders use the syntax of the Python |
| 89 |
``format()`` specification. A number of standard placeholders are supported |
| 90 |
(see the ``run`` documentation for a complete list): |
| 91 |
|
| 92 |
- ``{pwd}`` will be replaced with the full path of the current working directory |
| 93 |
- ``{dspath}`` will be replaced with the full path of the dataset that run is invoked on |
| 94 |
- ``{inputs}`` and ``{outputs}`` expand a space-separated list of the declared input and output paths |
| 95 |
|
| 96 |
Additionally, custom placeholders can be defined as configuration variables |
| 97 |
under the prefix ``datalad.run.substitutions.``. For example, a configuration |
| 98 |
setting ``datalad.run.substitutions.myfile=data.txt`` will cause the |
| 99 |
placeholder ``{myfile}`` to expand to ``data.txt``. |
| 100 |
|
| 101 |
Selection of individual items for placeholders that expand to multiple values |
| 102 |
is possible via the standard Python ``format()`` syntax, for example |
| 103 |
``{inputs[0]}``. |
| 104 |
|
| 105 |
|
| 106 |
Result records emitted by ``run`` |
| 107 |
================================= |
| 108 |
|
| 109 |
When performing a command execution ``run`` will emit results for: |
| 110 |
|
| 111 |
1. Input preparation (i.e. downloads) |
| 112 |
2. Output preparation (i.e. unlocks and removals) |
| 113 |
3. Command execution |
| 114 |
4. Dataset modification saving (i.e. additions, deletions, modifications) |
| 115 |
|
| 116 |
By default, ``run`` will stop on the first error. This means that, for example, |
| 117 |
any failure to download content will prevent command execution. A failing |
| 118 |
command will prevent saving a potential dataset modification. This behavior can |
| 119 |
be altered using the standard ``on_failure`` switch of the ``run`` command. |
| 120 |
|
| 121 |
The emitted result for the command execution contains the provenance record |
| 122 |
under the ``run_info`` key. |
| 123 |
|
| 124 |
|
| 125 |
Implementation details |
| 126 |
====================== |
| 127 |
|
| 128 |
Most of the described functionality is implemented by the function |
| 129 |
:func:`datalad.core.local.run.run_command`. It is interfaced by the ``run`` |
| 130 |
command, but also ``rerun``, a utility for automated re-execution based on |
| 131 |
provenance records, and ``containers-run`` (provided by the ``container`` |
| 132 |
extension package) for command execution in DataLad-tracked containerized |
| 133 |
environments. This function has a more complex interface, and supports a wider |
| 134 |
range of use cases than described here. |