Debian Codegraph @

sites/docs/concepts/configuration.rst

raw
1 .. _configuration:
2
3 =============
4 Configuration
5 =============
6
7 Introduction
8 ============
9
10 Invoke offers a multifaceted configuration mechanism allowing you to configure
11 both core behavior and that of your tasks, via a hierarchy of configuration
12 files, environment variables, :doc:`task namespaces </concepts/namespaces>` and
13 CLI flags.
14
15 The end result of configuration seeking, loading, parsing & merging is a
16 `.Config` object, which behaves like a (nested) Python dictionary. Invoke
17 references this object when it runs (determining the default behavior of
18 methods like `.Context.run`) and exposes it to users' tasks as
19 `.Context.config` or as shorthand attribute access on the `.Context` itself.
20
21
22 .. _config-hierarchy:
23
24 The configuration hierarchy
25 ===========================
26
27 In brief, the order in which configuration values override one another is as
28 follows:
29
30 #. **Internal default values** for behaviors which are controllable via
31 configuration. See :ref:`default-values` for details.
32 #. **Collection-driven configurations** defined in tasks modules via
33 `.Collection.configure`. (See :ref:`collection-configuration` below for
34 details.)
35
36 - Sub-collections' configurations get merged into the top level collection
37 and the final result forms the basis of the overall configuration setup.
38
39 #. **System-level configuration file** stored in ``/etc/``, such as
40 ``/etc/invoke.yaml``. (See :ref:`config-files` for details on this and the
41 other config-file entries.)
42 #. **User-level configuration file** found in the running user's home
43 directory, e.g. ``~/.invoke.yaml``.
44 #. **Project-level configuration file** living next to your top level
45 ``tasks.py``. For example, if your run of Invoke loads
46 ``/home/user/myproject/tasks.py`` (see our docs on :doc:`the load process
47 </concepts/loading>`), this might be ``/home/user/myproject/invoke.yaml``.
48 #. **Environment variables** found in the invoking shell environment.
49
50 - These aren't as strongly hierarchical as the rest, nor is the shell
51 environment namespace owned wholly by Invoke, so we must rely on slightly
52 verbose prefixing instead - see :ref:`env-vars` for details.
53
54 #. **Runtime configuration file** whose path is given to :option:`-f`, e.g.
55 ``inv -f /random/path/to/config_file.yaml``. This path may also be set via the
56 ``INVOKE_RUNTIME_CONFIG`` env var.
57 #. **Command-line flags** for certain core settings, such as :option:`-e`.
58 #. **Modifications made by user code** at runtime.
59
60
61 .. _default-values:
62
63 Default configuration values
64 ============================
65
66 Below is a list of all the configuration values and/or section Invoke itself
67 uses to control behaviors such as `.Context.run`'s ``echo`` and ``pty``
68 flags, task deduplication, and so forth.
69
70 .. note::
71 The storage location for these values is inside the `.Config` class,
72 specifically as the return value of `.Config.global_defaults`; see its API
73 docs for more details.
74
75 For convenience, we refer to nested setting names with a dotted syntax, so e.g.
76 ``foo.bar`` refers to what would be (in a Python config context) ``{'foo':
77 {'bar': <value here>}}``. Typically, these can be read or set on `.Config` and
78 `.Context` objects using attribute syntax, which looks nearly identical:
79 ``c.foo.bar``.
80
81 - The ``tasks`` config tree holds settings relating to task execution.
82
83 - ``tasks.dedupe`` controls :ref:`deduping` and defaults to ``True``. It
84 can also be overridden at runtime via :option:`--no-dedupe`.
85 - ``tasks.auto_dash_names`` controls whether task and collection names have
86 underscores turned to dashes on the CLI. Default: ``True``. See also
87 :ref:`dashes-vs-underscores`.
88 - ``tasks.collection_name`` controls the Python import name sought out by
89 :ref:`collection discovery <collection-discovery>`, and defaults to
90 ``"tasks"``.
91 - ``tasks.executor_class`` allows users to override the class instantiated
92 and used for task execution.
93
94 Must be a fully-qualified dotted path of the form
95 ``module(.submodule...).class``, where all but ``.class`` will be handed
96 to `importlib.import_module`, and ``class`` is expected to be an
97 attribute on that resulting module object.
98
99 Defaults to ``None``, meaning to use the running `.Program` object's
100 ``executor_class`` attribute.
101
102 .. warning::
103 Take care if using this setting in tandem with :ref:`custom program
104 binaries <reusing-as-a-binary>`, since custom programs may specify
105 their own default executor class (which your use of this setting will
106 override!) and assume certain behaviors stemming from that.
107
108 - ``tasks.ignore_unknown_help`` (default: ``False``) lets users disable
109 "help keys were supplied for nonexistent arguments" errors. Normally,
110 Invoke assumes such a situation implies a typo in the ``help`` argument
111 to ``@task``, but sometimes users have good reasons for this.
112 - ``tasks.search_root`` allows overriding the default :ref:`collection
113 discovery <collection-discovery>` root search location. It defaults to
114 ``None``, which indicates to use the executing process' current working
115 directory.
116
117 - The ``run`` tree controls the behavior of `.Runner.run`. Each member of this
118 tree (such as ``run.echo`` or ``run.pty``) maps directly to a `.Runner.run`
119 keyword argument of the same name; see that method's docstring for details on
120 what these settings do & what their default values are.
121 - The ``runners`` tree controls _which_ runner classes map to which execution
122 contexts; if you're using Invoke by itself, this will only tend to have a
123 single member, ``runners.local``. Client libraries may extend it with
124 additional key/value pairs, such as ``runners.remote``.
125 - The ``sudo`` tree controls the behavior of `.Context.sudo`:
126
127 - ``sudo.password`` controls the autoresponse password submitted to sudo's
128 password prompt. Default: ``None``.
129
130 .. warning::
131 While it's possible to store this setting, like any other, in
132 :doc:`configuration files </concepts/configuration>` -- doing so is
133 inherently insecure. We highly recommend filling this config value in
134 at runtime from a secrets management system of some kind.
135
136 - ``sudo.prompt`` holds the sudo password prompt text, which is both
137 supplied to ``sudo -p``, and searched for when performing
138 :doc:`auto-response </concepts/watchers>`. Default: ``[sudo] password:``.
139
140 - A top level config setting, ``debug``, controls whether debug-level output is
141 logged; it defaults to ``False``.
142
143 ``debug`` can be toggled via the :option:`-d` CLI flag, which enables
144 debugging after CLI parsing runs. It can also be toggled via the
145 ``INVOKE_DEBUG`` environment variable which - unlike regular env vars - is
146 honored from the start of execution and is thus useful for troubleshooting
147 parsing and/or config loading.
148
149 - A small config tree, ``timeouts``, holds various kinds of timeout controls.
150 At present, for Invoke, this only holds a ``command`` subkey, which controls
151 subprocess execution timeouts.
152
153 - Client code often adds more to this tree, and Invoke itself may add more
154 in the future as well.
155
156
157 .. _config-files:
158
159 Configuration files
160 ===================
161
162 Loading
163 -------
164
165 For each configuration file location mentioned in the previous section, we
166 search for files ending in ``.yaml``, ``.yml``, ``.json`` or ``.py`` (**in that
167 order!**), load the first one we find, and ignore any others that might exist.
168
169 For example, if Invoke is run on a system containing both ``/etc/invoke.yml``
170 *and* ``/etc/invoke.json``, **only the YAML file will be loaded**. This helps
171 keep things simple, both conceptually and in the implementation.
172
173 Format
174 ------
175
176 Invoke's configuration allows arbitrary nesting, and thus so do our config file
177 formats. All three of the below examples result in a configuration equivalent
178 to ``{'debug': True, 'run': {'echo': True}}``:
179
180 - **YAML**
181
182 .. code-block:: yaml
183
184 debug: true
185 run:
186 echo: true
187
188 - **JSON**
189
190 .. code-block:: javascript
191
192 {
193 "debug": true,
194 "run": {
195 "echo": true
196 }
197 }
198
199 - **Python**::
200
201 debug = True
202 run = {
203 "echo": True
204 }
205
206 For further details, see these languages' own documentation.
207
208
209 .. _env-vars:
210
211 Environment variables
212 =====================
213
214 Environment variables are a bit different from other configuration-setting
215 methods, since they don't provide a clean way to nest configuration keys, and
216 are also implicitly shared amongst the entire system's installed application
217 base.
218
219 In addition, due to implementation concerns, env vars must be pre-determined by
220 the levels below them in the config hierarchy (in other words - env vars may
221 only be used to override existing config values). If you need Invoke to
222 understand a ``FOOBAR`` environment variable, you must first declare a
223 ``foobar`` setting in a configuration file or in your task collections.
224
225 Basic rules
226 -----------
227
228 To mitigate the shell namespace problem, we simply prefix all our env vars with
229 ``INVOKE_``.
230
231 Nesting is performed via underscore separation, so a setting that looks like
232 e.g. ``{'run': {'echo': True}}`` at the Python level becomes
233 ``INVOKE_RUN_ECHO=1`` in a typical shell. See :ref:`env-var-nesting` below for
234 more on this.
235
236 Type casting
237 ------------
238
239 Since env vars can only be used to override existing settings, the previous
240 value of a given setting is used as a guide in casting the strings we get back
241 from the shell:
242
243 - If the current value is a Unicode string, it is replaced with the
244 value from the environment, with no casting whatsoever;
245 - If the current value is ``None``, it too is replaced with the string from the
246 environment;
247 - Booleans are set as follows: ``0`` and the empty value/string (e.g.
248 ``SETTING=``, or ``unset SETTING``, or etc) evaluate to ``False``, and any
249 other value evaluates to ``True``.
250 - Lists and tuples are currently unsupported and will raise an exception;
251
252 - In the future we may implement convenience transformations, such as
253 splitting on commas to form a list; however since users can always
254 perform such operations themselves, it may not be a high priority.
255
256 - All other types - integers, longs, floats, etc - are simply used as
257 constructors for the incoming value.
258
259 - For example, a ``foobar`` setting whose default value is the integer
260 ``1`` will run all env var inputs through `int`, and thus ``FOOBAR=5``
261 will result in the Python value ``5``, not ``"5"``.
262
263 .. _env-var-nesting:
264
265 Nesting vs underscored names
266 ----------------------------
267
268 Since environment variable keys are single strings, we must use some form of
269 string parsing to allow access to nested configuration settings. As mentioned
270 above, in basic use cases this just means using an underscore character:
271 ``{'run': {'echo': True}}`` becomes ``INVOKE_RUN_ECHO=1``.
272
273 However, ambiguity is introduced when the settings names themselves contain
274 underscores: is ``INVOKE_FOO_BAR=baz`` equivalent to ``{'foo': {'bar':
275 'baz'}}``, or to ``{'foo_bar': 'baz'}``? Thankfully, because env vars can only
276 be used to modify settings declared at the Python level or in config files, we
277 look at the current state of the config to determine the answer.
278
279 There is still a corner case where *both* possible interpretations exist as
280 valid config paths (e.g. ``{'foo': {'bar': 'default'}, 'foo_bar':
281 'otherdefault'}``). In this situation, we honor the `Zen of Python
282 <http://zen-of-python.info/in-the-face-of-ambiguity-refuse-the-temptation-to-guess.html#12>`_
283 and refuse to guess; an error is raised instead, counseling users to modify
284 their configuration layout or avoid using env vars for the setting in question.
285
286
287 .. _collection-configuration:
288
289 `.Collection`-based configuration
290 =================================
291
292 `.Collection` objects may contain a config mapping, set via
293 `.Collection.configure`, and (as per :ref:`the hierarchy <config-hierarchy>`)
294 this typically forms the lowest level of configuration in the system.
295
296 When collections are :doc:`nested </concepts/namespaces>`, configuration is
297 merged 'downwards' by default: when conflicts arise, outer namespaces closer to
298 the root will win, versus inner ones closer to the task being invoked.
299
300 .. note::
301 'Inner' tasks here are specifically those on the path from the root to the
302 one housing the invoked task. 'Sibling' subcollections are ignored.
303
304 A quick example of what this means::
305
306 from invoke import Collection, task
307
308 # This task & collection could just as easily come from
309 # another module somewhere.
310 @task
311 def mytask(c):
312 print(c['conflicted'])
313 inner = Collection('inner', mytask)
314 inner.configure({'conflicted': 'default value'})
315
316 # Our project's root namespace.
317 ns = Collection(inner)
318 ns.configure({'conflicted': 'override value'})
319
320 The result of calling ``inner.mytask``::
321
322 $ inv inner.mytask
323 override value
324
325
326 Example of real-world config use
327 ================================
328
329 The previous sections had small examples within them; this section provides a
330 more realistic-looking set of examples showing how the config system works.
331
332 Setup
333 -----
334
335 We'll start out with semi-realistic tasks that hardcode their values, and build
336 up to using the various configuration mechanisms. A small module for building
337 `Sphinx <http://sphinx-doc.org>`_ docs might begin like this::
338
339 from invoke import task
340
341 @task
342 def clean(c):
343 c.run("rm -rf docs/_build")
344
345 @task
346 def build(c):
347 c.run("sphinx-build docs docs/_build")
348
349 Then maybe you refactor the build target::
350
351 target = "docs/_build"
352
353 @task
354 def clean(c):
355 c.run("rm -rf {}".format(target))
356
357 @task
358 def build(c):
359 c.run("sphinx-build docs {}".format(target))
360
361 We can also allow runtime parameterization::
362
363 default_target = "docs/_build"
364
365 @task
366 def clean(c, target=default_target):
367 c.run("rm -rf {}".format(target))
368
369 @task
370 def build(c, target=default_target):
371 c.run("sphinx-build docs {}".format(target))
372
373 This task module works for a single set of users, but what if we want to allow
374 reuse? Somebody may want to use this module with a different default target.
375 Using the configuration data (made available via the context arg) to configure
376 these settings is usually the better solution [1]_.
377
378 Configuring via task collection
379 -------------------------------
380
381 The configuration `setting <.Collection.configure>` and `getting
382 <.Context.config>` APIs enable moving otherwise 'hardcoded' default values into
383 a config structure which downstream users are free to redefine. Let's apply
384 this to our example. First we add an explicit namespace object::
385
386 from invoke import Collection, task
387
388 default_target = "docs/_build"
389
390 @task
391 def clean(c, target=default_target):
392 c.run("rm -rf {}".format(target))
393
394 @task
395 def build(c, target=default_target):
396 c.run("sphinx-build docs {}".format(target))
397
398 ns = Collection(clean, build)
399
400 Then we can move the default build target value into the collection's default
401 configuration, and refer to it via the context. At this point we also change
402 our kwarg default value to be ``None`` so we can determine whether or not a
403 runtime value was given. The result::
404
405 @task
406 def clean(c, target=None):
407 if target is None:
408 target = c.sphinx.target
409 c.run("rm -rf {}".format(target))
410
411 @task
412 def build(c, target=None):
413 if target is None:
414 target = c.sphinx.target
415 c.run("sphinx-build docs {}".format(target))
416
417 ns = Collection(clean, build)
418 ns.configure({'sphinx': {'target': "docs/_build"}})
419
420 The result isn't significantly more complex than what we began with, and as
421 we'll see next, it's now trivial for users to override your defaults in various
422 ways.
423
424 Configuration overriding
425 ------------------------
426
427 The lowest-level override is, of course, just modifying the local `.Collection`
428 tree into which a distributed module has been imported. E.g. if the above
429 module is distributed as ``myproject.docs``, someone can define a ``tasks.py``
430 that does this::
431
432 from invoke import Collection, task
433 from myproject import docs
434
435 @task
436 def mylocaltask(c):
437 # Some local stuff goes here
438 pass
439
440 # Add 'docs' to our local root namespace, plus our own task
441 ns = Collection(mylocaltask, docs)
442
443 And then they can add this to the bottom::
444
445 # Our docs live in 'built_docs', not 'docs/_build'
446 ns.configure({'sphinx': {'target': "built_docs"}})
447
448 Now we have a ``docs`` sub-namespace whose build target defaults to
449 ``built_docs`` instead of ``docs/_build``. Runtime users can still override
450 this via flags (e.g. ``inv docs.build --target='some/other/dir'``) just as
451 before.
452
453 If you prefer configuration files over in-Python tweaking of your namespace
454 tree, that works just as well; instead of adding the line above to the previous
455 snippet, instead drop this into a file next to ``tasks.py`` named
456 ``invoke.yaml``::
457
458 sphinx:
459 target: built_docs
460
461 For this example, that sort of local-to-project conf file makes the most sense,
462 but don't forget that the :ref:`config hierarchy <config-hierarchy>` offers
463 additional configuration methods which may be suitable depending on your needs.
464
465
466 .. rubric:: Footnotes
467
468 .. [1]
469 Copying and modifying the file breaks code reuse; overriding the
470 module-level ``default_path`` variable won't play well with concurrency;
471 wrapping the tasks with different default arguments works but is fragile
472 and adds boilerplate.