Debian Codegraph @

docs/source/design/progress_reporting.rst

raw
1 .. -*- mode: rst -*-
2 .. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79:
3
4 .. _chap_design_progress_reporting:
5
6 ******************
7 Progress reporting
8 ******************
9
10 .. topic:: Specification scope and status
11
12 This specification describes the current implementation.
13
14
15 Progress reporting is implemented via the logging system. A dedicated function
16 :py:func:`datalad.log.log_progress` represents the main API for progress
17 reporting. For some standard use cases, the utilities
18 :py:func:`datalad.log.with_progress` and
19 :py:func:`datalad.log.with_result_progress` can simplify result reporting
20 further.
21
22
23 Design and implementation
24 =========================
25
26 This basic idea is to use an instance of datalad's loggers to emit log messages
27 with particular attributes that are picked up by
28 :py:class:`datalad.log.ProgressHandler` (derived from
29 :py:class:`logging.Handler`), and are acted on differently, depending on
30 configuration and conditions of a session (e.g., interactive terminal sessions
31 vs. non-interactive usage in scripts). This variable behavior is implemented
32 via the use of :py:mod:`logging` standard library log filters and handlers.
33 Roughly speaking, :py:class:`datalad.log.ProgressHandler` will only be used for
34 interactive sessions. In non-interactive cases, progress log messages are
35 inspected by :py:func:`datalad.log.filter_noninteractive_progress`, and are
36 either discarded or treated like any other log message (see
37 :py:meth:`datalad.log.LoggerHelper.get_initialized_logger` for details on the
38 handler and filter setup).
39
40 :py:class:`datalad.log.ProgressHandler` inspects incoming log records for
41 attributes with names starting with `dlm_progress`. It will only process such
42 records and pass others on to the underlying original log handler otherwise.
43
44 :py:class:`datalad.log.ProgressHandler` takes care of creating, updating and
45 destroying any number of simultaneously running progress bars. Progress reports
46 must identify the respective process via an arbitrary string ID. It is the
47 caller's responsibility to ensure that this ID is unique to the target
48 process/activity.
49
50
51 Reporting progress with `log_progress()`
52 ========================================
53
54 Typical progress reporting via :py:func:`datalad.log.log_progress` involves
55 three types of calls.
56
57 1. Start reporting progress about a process
58 -------------------------------------------
59
60 A typical call to start of progress reporting looks like this
61
62 .. code-block:: python
63
64 log_progress(
65 # the callable used to emit log messages
66 lgr.info,
67 # a unique identifiers of the activity progress is reported for
68 identifier,
69 # main message
70 'Unlocking files',
71 # optional unit string for a progress bar
72 unit=' Files',
73 # optional label to be displayed in a progress bar
74 label='Unlocking',
75 # maximum value for a progress bar
76 total=nfiles,
77 )
78
79 A new progress bar will be created automatically for any report with a previously
80 unseen activity ``identifier``. It can be configured via the specification of
81 a number of arguments, most notably a target ``total`` for the progress bar.
82 See :py:func:`datalad.log.log_progress` for a complete overview.
83
84 Starting a progress report must be done with a dedicated call. It cannot be combined
85 with a progress update.
86
87
88 2. Update progress information about a process
89 ----------------------------------------------
90
91 Any subsequent call to :py:func:`datalad.log.log_progress` with an activity
92 identifier that has already been seen either updates, or finishes the progress
93 reporting for an activity. Updates must contain an ``update`` key which either
94 specifies a new value (if `increment=False`, the default) or an increment to
95 previously known value (if `increment=True`):
96
97 .. code-block:: python
98
99 log_progress(
100 lgr.info,
101 # must match the identifier used to start the progress reporting
102 identifier,
103 # arbitrary message content, string expansion supported just like
104 # regular log messages
105 "Files to unlock %i", nfiles,
106 # critical key for report updates
107 update=1,
108 # ``update`` could be an absolute value or an increment
109 increment=True
110 )
111
112 Updating a progress report can only be done after a progress reporting was
113 initialized (see above).
114
115
116 3. Report completion of a process
117 ---------------------------------
118
119 A progress bar will remain active until it is explicitly taken down, even if an
120 initially declared ``total`` value may have been reached. Finishing a progress
121 report requires a final log message with the corresponding identifiers which,
122 like the first initializing message, does NOT contain an ``update`` key.
123
124 .. code-block:: python
125
126 log_progress(
127 lgr.info,
128 identifier,
129 # closing log message
130 "Completed unlocking files",
131 )
132
133
134 Progress reporting in non-interactive sessions
135 ----------------------------------------------
136
137 :py:func:`datalad.log.log_progress` takes a `noninteractive_level` argument
138 that can be used to specify a log level at which progress is logged when no
139 progress bars can be used, but actual log messages are produced.
140
141 .. code-block:: python
142
143 import logging
144
145 log_progress(
146 lgr.info,
147 identifier,
148 "Completed unlocking files",
149 noninteractive_level=logging.INFO
150 )
151
152 Each call to :py:func:`~datalad.log.log_progress` can be given a different
153 log level, in order to control the verbosity of the reporting in such a scenario.
154 For example, it is possible to log the start or end of an activity at a higher
155 level than intermediate updates. It is also possible to single out particular
156 intermediate events, and report them at a higher level.
157
158 If no `noninteractive_level` is specified, the progress update is unconditionally
159 logged at the level implied by the given logger callable.
160
161
162 Reporting progress with `with_(result_)progress()`
163 ==================================================
164
165 For cases were a list of items needs to be processes sequentially, and progress
166 shall be communicated, two additional helpers could be used: the decorators
167 :py:func:`datalad.log.with_progress` and
168 :py:func:`datalad.log.with_result_progress`. They require a callable that takes
169 a list (or more generally a sequence) of items to be processed as the first
170 positional argument. They both set up and perform all necessary calls to
171 :py:func:`~datalad.log.log_progress`.
172
173 The difference between these helpers is that
174 :py:func:`datalad.log.with_result_progress` expects a callable to produce
175 DataLad result records, and supports customs filters to decide which particular
176 result records to consider for progress reporting (e.g., only records for a
177 particular `action` and `type`).
178
179
180 Output non-progress information without interfering with progress bars
181 ======================================================================
182
183 :py:func:`~datalad.log.log_progress` can also be useful when not reporting
184 progress, but ensuring that no other output is interfering with progress bars,
185 and vice versa. The argument `maint` can be used in this case, with no
186 particular activity identifier (it always impacts all active progress bars):
187
188
189 .. code-block:: python
190
191 log_progress(
192 lgr.info,
193 None,
194 'Clear progress bars',
195 maint='clear',
196 )
197
198
199 This call will trigger a temporary discontinuation of any progress bar display.
200 Progress bars can either be re-enabled all at once, by an analog message with
201 ``maint='refresh'``, or will re-show themselves automatically when the next
202 update is received. A :py:func:`~datalad.log.no_progress` context manager helper
203 can be used to surround your context with those two calls to prevent progress
204 bars from interfering.