| 1 |
.. -*- mode: rst -*- |
| 2 |
.. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79: |
| 3 |
|
| 4 |
.. _chap_design_progress_reporting: |
| 5 |
|
| 6 |
****************** |
| 7 |
Progress reporting |
| 8 |
****************** |
| 9 |
|
| 10 |
.. topic:: Specification scope and status |
| 11 |
|
| 12 |
This specification describes the current implementation. |
| 13 |
|
| 14 |
|
| 15 |
Progress reporting is implemented via the logging system. A dedicated function |
| 16 |
:py:func:`datalad.log.log_progress` represents the main API for progress |
| 17 |
reporting. For some standard use cases, the utilities |
| 18 |
:py:func:`datalad.log.with_progress` and |
| 19 |
:py:func:`datalad.log.with_result_progress` can simplify result reporting |
| 20 |
further. |
| 21 |
|
| 22 |
|
| 23 |
Design and implementation |
| 24 |
========================= |
| 25 |
|
| 26 |
This basic idea is to use an instance of datalad's loggers to emit log messages |
| 27 |
with particular attributes that are picked up by |
| 28 |
:py:class:`datalad.log.ProgressHandler` (derived from |
| 29 |
:py:class:`logging.Handler`), and are acted on differently, depending on |
| 30 |
configuration and conditions of a session (e.g., interactive terminal sessions |
| 31 |
vs. non-interactive usage in scripts). This variable behavior is implemented |
| 32 |
via the use of :py:mod:`logging` standard library log filters and handlers. |
| 33 |
Roughly speaking, :py:class:`datalad.log.ProgressHandler` will only be used for |
| 34 |
interactive sessions. In non-interactive cases, progress log messages are |
| 35 |
inspected by :py:func:`datalad.log.filter_noninteractive_progress`, and are |
| 36 |
either discarded or treated like any other log message (see |
| 37 |
:py:meth:`datalad.log.LoggerHelper.get_initialized_logger` for details on the |
| 38 |
handler and filter setup). |
| 39 |
|
| 40 |
:py:class:`datalad.log.ProgressHandler` inspects incoming log records for |
| 41 |
attributes with names starting with `dlm_progress`. It will only process such |
| 42 |
records and pass others on to the underlying original log handler otherwise. |
| 43 |
|
| 44 |
:py:class:`datalad.log.ProgressHandler` takes care of creating, updating and |
| 45 |
destroying any number of simultaneously running progress bars. Progress reports |
| 46 |
must identify the respective process via an arbitrary string ID. It is the |
| 47 |
caller's responsibility to ensure that this ID is unique to the target |
| 48 |
process/activity. |
| 49 |
|
| 50 |
|
| 51 |
Reporting progress with `log_progress()` |
| 52 |
======================================== |
| 53 |
|
| 54 |
Typical progress reporting via :py:func:`datalad.log.log_progress` involves |
| 55 |
three types of calls. |
| 56 |
|
| 57 |
1. Start reporting progress about a process |
| 58 |
------------------------------------------- |
| 59 |
|
| 60 |
A typical call to start of progress reporting looks like this |
| 61 |
|
| 62 |
.. code-block:: python |
| 63 |
|
| 64 |
log_progress( |
| 65 |
# the callable used to emit log messages |
| 66 |
lgr.info, |
| 67 |
# a unique identifiers of the activity progress is reported for |
| 68 |
identifier, |
| 69 |
# main message |
| 70 |
'Unlocking files', |
| 71 |
# optional unit string for a progress bar |
| 72 |
unit=' Files', |
| 73 |
# optional label to be displayed in a progress bar |
| 74 |
label='Unlocking', |
| 75 |
# maximum value for a progress bar |
| 76 |
total=nfiles, |
| 77 |
) |
| 78 |
|
| 79 |
A new progress bar will be created automatically for any report with a previously |
| 80 |
unseen activity ``identifier``. It can be configured via the specification of |
| 81 |
a number of arguments, most notably a target ``total`` for the progress bar. |
| 82 |
See :py:func:`datalad.log.log_progress` for a complete overview. |
| 83 |
|
| 84 |
Starting a progress report must be done with a dedicated call. It cannot be combined |
| 85 |
with a progress update. |
| 86 |
|
| 87 |
|
| 88 |
2. Update progress information about a process |
| 89 |
---------------------------------------------- |
| 90 |
|
| 91 |
Any subsequent call to :py:func:`datalad.log.log_progress` with an activity |
| 92 |
identifier that has already been seen either updates, or finishes the progress |
| 93 |
reporting for an activity. Updates must contain an ``update`` key which either |
| 94 |
specifies a new value (if `increment=False`, the default) or an increment to |
| 95 |
previously known value (if `increment=True`): |
| 96 |
|
| 97 |
.. code-block:: python |
| 98 |
|
| 99 |
log_progress( |
| 100 |
lgr.info, |
| 101 |
# must match the identifier used to start the progress reporting |
| 102 |
identifier, |
| 103 |
# arbitrary message content, string expansion supported just like |
| 104 |
# regular log messages |
| 105 |
"Files to unlock %i", nfiles, |
| 106 |
# critical key for report updates |
| 107 |
update=1, |
| 108 |
# ``update`` could be an absolute value or an increment |
| 109 |
increment=True |
| 110 |
) |
| 111 |
|
| 112 |
Updating a progress report can only be done after a progress reporting was |
| 113 |
initialized (see above). |
| 114 |
|
| 115 |
|
| 116 |
3. Report completion of a process |
| 117 |
--------------------------------- |
| 118 |
|
| 119 |
A progress bar will remain active until it is explicitly taken down, even if an |
| 120 |
initially declared ``total`` value may have been reached. Finishing a progress |
| 121 |
report requires a final log message with the corresponding identifiers which, |
| 122 |
like the first initializing message, does NOT contain an ``update`` key. |
| 123 |
|
| 124 |
.. code-block:: python |
| 125 |
|
| 126 |
log_progress( |
| 127 |
lgr.info, |
| 128 |
identifier, |
| 129 |
# closing log message |
| 130 |
"Completed unlocking files", |
| 131 |
) |
| 132 |
|
| 133 |
|
| 134 |
Progress reporting in non-interactive sessions |
| 135 |
---------------------------------------------- |
| 136 |
|
| 137 |
:py:func:`datalad.log.log_progress` takes a `noninteractive_level` argument |
| 138 |
that can be used to specify a log level at which progress is logged when no |
| 139 |
progress bars can be used, but actual log messages are produced. |
| 140 |
|
| 141 |
.. code-block:: python |
| 142 |
|
| 143 |
import logging |
| 144 |
|
| 145 |
log_progress( |
| 146 |
lgr.info, |
| 147 |
identifier, |
| 148 |
"Completed unlocking files", |
| 149 |
noninteractive_level=logging.INFO |
| 150 |
) |
| 151 |
|
| 152 |
Each call to :py:func:`~datalad.log.log_progress` can be given a different |
| 153 |
log level, in order to control the verbosity of the reporting in such a scenario. |
| 154 |
For example, it is possible to log the start or end of an activity at a higher |
| 155 |
level than intermediate updates. It is also possible to single out particular |
| 156 |
intermediate events, and report them at a higher level. |
| 157 |
|
| 158 |
If no `noninteractive_level` is specified, the progress update is unconditionally |
| 159 |
logged at the level implied by the given logger callable. |
| 160 |
|
| 161 |
|
| 162 |
Reporting progress with `with_(result_)progress()` |
| 163 |
================================================== |
| 164 |
|
| 165 |
For cases were a list of items needs to be processes sequentially, and progress |
| 166 |
shall be communicated, two additional helpers could be used: the decorators |
| 167 |
:py:func:`datalad.log.with_progress` and |
| 168 |
:py:func:`datalad.log.with_result_progress`. They require a callable that takes |
| 169 |
a list (or more generally a sequence) of items to be processed as the first |
| 170 |
positional argument. They both set up and perform all necessary calls to |
| 171 |
:py:func:`~datalad.log.log_progress`. |
| 172 |
|
| 173 |
The difference between these helpers is that |
| 174 |
:py:func:`datalad.log.with_result_progress` expects a callable to produce |
| 175 |
DataLad result records, and supports customs filters to decide which particular |
| 176 |
result records to consider for progress reporting (e.g., only records for a |
| 177 |
particular `action` and `type`). |
| 178 |
|
| 179 |
|
| 180 |
Output non-progress information without interfering with progress bars |
| 181 |
====================================================================== |
| 182 |
|
| 183 |
:py:func:`~datalad.log.log_progress` can also be useful when not reporting |
| 184 |
progress, but ensuring that no other output is interfering with progress bars, |
| 185 |
and vice versa. The argument `maint` can be used in this case, with no |
| 186 |
particular activity identifier (it always impacts all active progress bars): |
| 187 |
|
| 188 |
|
| 189 |
.. code-block:: python |
| 190 |
|
| 191 |
log_progress( |
| 192 |
lgr.info, |
| 193 |
None, |
| 194 |
'Clear progress bars', |
| 195 |
maint='clear', |
| 196 |
) |
| 197 |
|
| 198 |
|
| 199 |
This call will trigger a temporary discontinuation of any progress bar display. |
| 200 |
Progress bars can either be re-enabled all at once, by an analog message with |
| 201 |
``maint='refresh'``, or will re-show themselves automatically when the next |
| 202 |
update is received. A :py:func:`~datalad.log.no_progress` context manager helper |
| 203 |
can be used to surround your context with those two calls to prevent progress |
| 204 |
bars from interfering. |