| 1 |
.. _configuration: |
| 2 |
|
| 3 |
============= |
| 4 |
Configuration |
| 5 |
============= |
| 6 |
|
| 7 |
Introduction |
| 8 |
============ |
| 9 |
|
| 10 |
Invoke offers a multifaceted configuration mechanism allowing you to configure |
| 11 |
both core behavior and that of your tasks, via a hierarchy of configuration |
| 12 |
files, environment variables, :doc:`task namespaces </concepts/namespaces>` and |
| 13 |
CLI flags. |
| 14 |
|
| 15 |
The end result of configuration seeking, loading, parsing & merging is a |
| 16 |
`.Config` object, which behaves like a (nested) Python dictionary. Invoke |
| 17 |
references this object when it runs (determining the default behavior of |
| 18 |
methods like `.Context.run`) and exposes it to users' tasks as |
| 19 |
`.Context.config` or as shorthand attribute access on the `.Context` itself. |
| 20 |
|
| 21 |
|
| 22 |
.. _config-hierarchy: |
| 23 |
|
| 24 |
The configuration hierarchy |
| 25 |
=========================== |
| 26 |
|
| 27 |
In brief, the order in which configuration values override one another is as |
| 28 |
follows: |
| 29 |
|
| 30 |
#. **Internal default values** for behaviors which are controllable via |
| 31 |
configuration. See :ref:`default-values` for details. |
| 32 |
#. **Collection-driven configurations** defined in tasks modules via |
| 33 |
`.Collection.configure`. (See :ref:`collection-configuration` below for |
| 34 |
details.) |
| 35 |
|
| 36 |
- Sub-collections' configurations get merged into the top level collection |
| 37 |
and the final result forms the basis of the overall configuration setup. |
| 38 |
|
| 39 |
#. **System-level configuration file** stored in ``/etc/``, such as |
| 40 |
``/etc/invoke.yaml``. (See :ref:`config-files` for details on this and the |
| 41 |
other config-file entries.) |
| 42 |
#. **User-level configuration file** found in the running user's home |
| 43 |
directory, e.g. ``~/.invoke.yaml``. |
| 44 |
#. **Project-level configuration file** living next to your top level |
| 45 |
``tasks.py``. For example, if your run of Invoke loads |
| 46 |
``/home/user/myproject/tasks.py`` (see our docs on :doc:`the load process |
| 47 |
</concepts/loading>`), this might be ``/home/user/myproject/invoke.yaml``. |
| 48 |
#. **Environment variables** found in the invoking shell environment. |
| 49 |
|
| 50 |
- These aren't as strongly hierarchical as the rest, nor is the shell |
| 51 |
environment namespace owned wholly by Invoke, so we must rely on slightly |
| 52 |
verbose prefixing instead - see :ref:`env-vars` for details. |
| 53 |
|
| 54 |
#. **Runtime configuration file** whose path is given to :option:`-f`, e.g. |
| 55 |
``inv -f /random/path/to/config_file.yaml``. This path may also be set via the |
| 56 |
``INVOKE_RUNTIME_CONFIG`` env var. |
| 57 |
#. **Command-line flags** for certain core settings, such as :option:`-e`. |
| 58 |
#. **Modifications made by user code** at runtime. |
| 59 |
|
| 60 |
|
| 61 |
.. _default-values: |
| 62 |
|
| 63 |
Default configuration values |
| 64 |
============================ |
| 65 |
|
| 66 |
Below is a list of all the configuration values and/or section Invoke itself |
| 67 |
uses to control behaviors such as `.Context.run`'s ``echo`` and ``pty`` |
| 68 |
flags, task deduplication, and so forth. |
| 69 |
|
| 70 |
.. note:: |
| 71 |
The storage location for these values is inside the `.Config` class, |
| 72 |
specifically as the return value of `.Config.global_defaults`; see its API |
| 73 |
docs for more details. |
| 74 |
|
| 75 |
For convenience, we refer to nested setting names with a dotted syntax, so e.g. |
| 76 |
``foo.bar`` refers to what would be (in a Python config context) ``{'foo': |
| 77 |
{'bar': <value here>}}``. Typically, these can be read or set on `.Config` and |
| 78 |
`.Context` objects using attribute syntax, which looks nearly identical: |
| 79 |
``c.foo.bar``. |
| 80 |
|
| 81 |
- The ``tasks`` config tree holds settings relating to task execution. |
| 82 |
|
| 83 |
- ``tasks.dedupe`` controls :ref:`deduping` and defaults to ``True``. It |
| 84 |
can also be overridden at runtime via :option:`--no-dedupe`. |
| 85 |
- ``tasks.auto_dash_names`` controls whether task and collection names have |
| 86 |
underscores turned to dashes on the CLI. Default: ``True``. See also |
| 87 |
:ref:`dashes-vs-underscores`. |
| 88 |
- ``tasks.collection_name`` controls the Python import name sought out by |
| 89 |
:ref:`collection discovery <collection-discovery>`, and defaults to |
| 90 |
``"tasks"``. |
| 91 |
- ``tasks.executor_class`` allows users to override the class instantiated |
| 92 |
and used for task execution. |
| 93 |
|
| 94 |
Must be a fully-qualified dotted path of the form |
| 95 |
``module(.submodule...).class``, where all but ``.class`` will be handed |
| 96 |
to `importlib.import_module`, and ``class`` is expected to be an |
| 97 |
attribute on that resulting module object. |
| 98 |
|
| 99 |
Defaults to ``None``, meaning to use the running `.Program` object's |
| 100 |
``executor_class`` attribute. |
| 101 |
|
| 102 |
.. warning:: |
| 103 |
Take care if using this setting in tandem with :ref:`custom program |
| 104 |
binaries <reusing-as-a-binary>`, since custom programs may specify |
| 105 |
their own default executor class (which your use of this setting will |
| 106 |
override!) and assume certain behaviors stemming from that. |
| 107 |
|
| 108 |
- ``tasks.ignore_unknown_help`` (default: ``False``) lets users disable |
| 109 |
"help keys were supplied for nonexistent arguments" errors. Normally, |
| 110 |
Invoke assumes such a situation implies a typo in the ``help`` argument |
| 111 |
to ``@task``, but sometimes users have good reasons for this. |
| 112 |
- ``tasks.search_root`` allows overriding the default :ref:`collection |
| 113 |
discovery <collection-discovery>` root search location. It defaults to |
| 114 |
``None``, which indicates to use the executing process' current working |
| 115 |
directory. |
| 116 |
|
| 117 |
- The ``run`` tree controls the behavior of `.Runner.run`. Each member of this |
| 118 |
tree (such as ``run.echo`` or ``run.pty``) maps directly to a `.Runner.run` |
| 119 |
keyword argument of the same name; see that method's docstring for details on |
| 120 |
what these settings do & what their default values are. |
| 121 |
- The ``runners`` tree controls _which_ runner classes map to which execution |
| 122 |
contexts; if you're using Invoke by itself, this will only tend to have a |
| 123 |
single member, ``runners.local``. Client libraries may extend it with |
| 124 |
additional key/value pairs, such as ``runners.remote``. |
| 125 |
- The ``sudo`` tree controls the behavior of `.Context.sudo`: |
| 126 |
|
| 127 |
- ``sudo.password`` controls the autoresponse password submitted to sudo's |
| 128 |
password prompt. Default: ``None``. |
| 129 |
|
| 130 |
.. warning:: |
| 131 |
While it's possible to store this setting, like any other, in |
| 132 |
:doc:`configuration files </concepts/configuration>` -- doing so is |
| 133 |
inherently insecure. We highly recommend filling this config value in |
| 134 |
at runtime from a secrets management system of some kind. |
| 135 |
|
| 136 |
- ``sudo.prompt`` holds the sudo password prompt text, which is both |
| 137 |
supplied to ``sudo -p``, and searched for when performing |
| 138 |
:doc:`auto-response </concepts/watchers>`. Default: ``[sudo] password:``. |
| 139 |
|
| 140 |
- A top level config setting, ``debug``, controls whether debug-level output is |
| 141 |
logged; it defaults to ``False``. |
| 142 |
|
| 143 |
``debug`` can be toggled via the :option:`-d` CLI flag, which enables |
| 144 |
debugging after CLI parsing runs. It can also be toggled via the |
| 145 |
``INVOKE_DEBUG`` environment variable which - unlike regular env vars - is |
| 146 |
honored from the start of execution and is thus useful for troubleshooting |
| 147 |
parsing and/or config loading. |
| 148 |
|
| 149 |
- A small config tree, ``timeouts``, holds various kinds of timeout controls. |
| 150 |
At present, for Invoke, this only holds a ``command`` subkey, which controls |
| 151 |
subprocess execution timeouts. |
| 152 |
|
| 153 |
- Client code often adds more to this tree, and Invoke itself may add more |
| 154 |
in the future as well. |
| 155 |
|
| 156 |
|
| 157 |
.. _config-files: |
| 158 |
|
| 159 |
Configuration files |
| 160 |
=================== |
| 161 |
|
| 162 |
Loading |
| 163 |
------- |
| 164 |
|
| 165 |
For each configuration file location mentioned in the previous section, we |
| 166 |
search for files ending in ``.yaml``, ``.yml``, ``.json`` or ``.py`` (**in that |
| 167 |
order!**), load the first one we find, and ignore any others that might exist. |
| 168 |
|
| 169 |
For example, if Invoke is run on a system containing both ``/etc/invoke.yml`` |
| 170 |
*and* ``/etc/invoke.json``, **only the YAML file will be loaded**. This helps |
| 171 |
keep things simple, both conceptually and in the implementation. |
| 172 |
|
| 173 |
Format |
| 174 |
------ |
| 175 |
|
| 176 |
Invoke's configuration allows arbitrary nesting, and thus so do our config file |
| 177 |
formats. All three of the below examples result in a configuration equivalent |
| 178 |
to ``{'debug': True, 'run': {'echo': True}}``: |
| 179 |
|
| 180 |
- **YAML** |
| 181 |
|
| 182 |
.. code-block:: yaml |
| 183 |
|
| 184 |
debug: true |
| 185 |
run: |
| 186 |
echo: true |
| 187 |
|
| 188 |
- **JSON** |
| 189 |
|
| 190 |
.. code-block:: javascript |
| 191 |
|
| 192 |
{ |
| 193 |
"debug": true, |
| 194 |
"run": { |
| 195 |
"echo": true |
| 196 |
} |
| 197 |
} |
| 198 |
|
| 199 |
- **Python**:: |
| 200 |
|
| 201 |
debug = True |
| 202 |
run = { |
| 203 |
"echo": True |
| 204 |
} |
| 205 |
|
| 206 |
For further details, see these languages' own documentation. |
| 207 |
|
| 208 |
|
| 209 |
.. _env-vars: |
| 210 |
|
| 211 |
Environment variables |
| 212 |
===================== |
| 213 |
|
| 214 |
Environment variables are a bit different from other configuration-setting |
| 215 |
methods, since they don't provide a clean way to nest configuration keys, and |
| 216 |
are also implicitly shared amongst the entire system's installed application |
| 217 |
base. |
| 218 |
|
| 219 |
In addition, due to implementation concerns, env vars must be pre-determined by |
| 220 |
the levels below them in the config hierarchy (in other words - env vars may |
| 221 |
only be used to override existing config values). If you need Invoke to |
| 222 |
understand a ``FOOBAR`` environment variable, you must first declare a |
| 223 |
``foobar`` setting in a configuration file or in your task collections. |
| 224 |
|
| 225 |
Basic rules |
| 226 |
----------- |
| 227 |
|
| 228 |
To mitigate the shell namespace problem, we simply prefix all our env vars with |
| 229 |
``INVOKE_``. |
| 230 |
|
| 231 |
Nesting is performed via underscore separation, so a setting that looks like |
| 232 |
e.g. ``{'run': {'echo': True}}`` at the Python level becomes |
| 233 |
``INVOKE_RUN_ECHO=1`` in a typical shell. See :ref:`env-var-nesting` below for |
| 234 |
more on this. |
| 235 |
|
| 236 |
Type casting |
| 237 |
------------ |
| 238 |
|
| 239 |
Since env vars can only be used to override existing settings, the previous |
| 240 |
value of a given setting is used as a guide in casting the strings we get back |
| 241 |
from the shell: |
| 242 |
|
| 243 |
- If the current value is a Unicode string, it is replaced with the |
| 244 |
value from the environment, with no casting whatsoever; |
| 245 |
- If the current value is ``None``, it too is replaced with the string from the |
| 246 |
environment; |
| 247 |
- Booleans are set as follows: ``0`` and the empty value/string (e.g. |
| 248 |
``SETTING=``, or ``unset SETTING``, or etc) evaluate to ``False``, and any |
| 249 |
other value evaluates to ``True``. |
| 250 |
- Lists and tuples are currently unsupported and will raise an exception; |
| 251 |
|
| 252 |
- In the future we may implement convenience transformations, such as |
| 253 |
splitting on commas to form a list; however since users can always |
| 254 |
perform such operations themselves, it may not be a high priority. |
| 255 |
|
| 256 |
- All other types - integers, longs, floats, etc - are simply used as |
| 257 |
constructors for the incoming value. |
| 258 |
|
| 259 |
- For example, a ``foobar`` setting whose default value is the integer |
| 260 |
``1`` will run all env var inputs through `int`, and thus ``FOOBAR=5`` |
| 261 |
will result in the Python value ``5``, not ``"5"``. |
| 262 |
|
| 263 |
.. _env-var-nesting: |
| 264 |
|
| 265 |
Nesting vs underscored names |
| 266 |
---------------------------- |
| 267 |
|
| 268 |
Since environment variable keys are single strings, we must use some form of |
| 269 |
string parsing to allow access to nested configuration settings. As mentioned |
| 270 |
above, in basic use cases this just means using an underscore character: |
| 271 |
``{'run': {'echo': True}}`` becomes ``INVOKE_RUN_ECHO=1``. |
| 272 |
|
| 273 |
However, ambiguity is introduced when the settings names themselves contain |
| 274 |
underscores: is ``INVOKE_FOO_BAR=baz`` equivalent to ``{'foo': {'bar': |
| 275 |
'baz'}}``, or to ``{'foo_bar': 'baz'}``? Thankfully, because env vars can only |
| 276 |
be used to modify settings declared at the Python level or in config files, we |
| 277 |
look at the current state of the config to determine the answer. |
| 278 |
|
| 279 |
There is still a corner case where *both* possible interpretations exist as |
| 280 |
valid config paths (e.g. ``{'foo': {'bar': 'default'}, 'foo_bar': |
| 281 |
'otherdefault'}``). In this situation, we honor the `Zen of Python |
| 282 |
<http://zen-of-python.info/in-the-face-of-ambiguity-refuse-the-temptation-to-guess.html#12>`_ |
| 283 |
and refuse to guess; an error is raised instead, counseling users to modify |
| 284 |
their configuration layout or avoid using env vars for the setting in question. |
| 285 |
|
| 286 |
|
| 287 |
.. _collection-configuration: |
| 288 |
|
| 289 |
`.Collection`-based configuration |
| 290 |
================================= |
| 291 |
|
| 292 |
`.Collection` objects may contain a config mapping, set via |
| 293 |
`.Collection.configure`, and (as per :ref:`the hierarchy <config-hierarchy>`) |
| 294 |
this typically forms the lowest level of configuration in the system. |
| 295 |
|
| 296 |
When collections are :doc:`nested </concepts/namespaces>`, configuration is |
| 297 |
merged 'downwards' by default: when conflicts arise, outer namespaces closer to |
| 298 |
the root will win, versus inner ones closer to the task being invoked. |
| 299 |
|
| 300 |
.. note:: |
| 301 |
'Inner' tasks here are specifically those on the path from the root to the |
| 302 |
one housing the invoked task. 'Sibling' subcollections are ignored. |
| 303 |
|
| 304 |
A quick example of what this means:: |
| 305 |
|
| 306 |
from invoke import Collection, task |
| 307 |
|
| 308 |
# This task & collection could just as easily come from |
| 309 |
# another module somewhere. |
| 310 |
@task |
| 311 |
def mytask(c): |
| 312 |
print(c['conflicted']) |
| 313 |
inner = Collection('inner', mytask) |
| 314 |
inner.configure({'conflicted': 'default value'}) |
| 315 |
|
| 316 |
# Our project's root namespace. |
| 317 |
ns = Collection(inner) |
| 318 |
ns.configure({'conflicted': 'override value'}) |
| 319 |
|
| 320 |
The result of calling ``inner.mytask``:: |
| 321 |
|
| 322 |
$ inv inner.mytask |
| 323 |
override value |
| 324 |
|
| 325 |
|
| 326 |
Example of real-world config use |
| 327 |
================================ |
| 328 |
|
| 329 |
The previous sections had small examples within them; this section provides a |
| 330 |
more realistic-looking set of examples showing how the config system works. |
| 331 |
|
| 332 |
Setup |
| 333 |
----- |
| 334 |
|
| 335 |
We'll start out with semi-realistic tasks that hardcode their values, and build |
| 336 |
up to using the various configuration mechanisms. A small module for building |
| 337 |
`Sphinx <http://sphinx-doc.org>`_ docs might begin like this:: |
| 338 |
|
| 339 |
from invoke import task |
| 340 |
|
| 341 |
@task |
| 342 |
def clean(c): |
| 343 |
c.run("rm -rf docs/_build") |
| 344 |
|
| 345 |
@task |
| 346 |
def build(c): |
| 347 |
c.run("sphinx-build docs docs/_build") |
| 348 |
|
| 349 |
Then maybe you refactor the build target:: |
| 350 |
|
| 351 |
target = "docs/_build" |
| 352 |
|
| 353 |
@task |
| 354 |
def clean(c): |
| 355 |
c.run("rm -rf {}".format(target)) |
| 356 |
|
| 357 |
@task |
| 358 |
def build(c): |
| 359 |
c.run("sphinx-build docs {}".format(target)) |
| 360 |
|
| 361 |
We can also allow runtime parameterization:: |
| 362 |
|
| 363 |
default_target = "docs/_build" |
| 364 |
|
| 365 |
@task |
| 366 |
def clean(c, target=default_target): |
| 367 |
c.run("rm -rf {}".format(target)) |
| 368 |
|
| 369 |
@task |
| 370 |
def build(c, target=default_target): |
| 371 |
c.run("sphinx-build docs {}".format(target)) |
| 372 |
|
| 373 |
This task module works for a single set of users, but what if we want to allow |
| 374 |
reuse? Somebody may want to use this module with a different default target. |
| 375 |
Using the configuration data (made available via the context arg) to configure |
| 376 |
these settings is usually the better solution [1]_. |
| 377 |
|
| 378 |
Configuring via task collection |
| 379 |
------------------------------- |
| 380 |
|
| 381 |
The configuration `setting <.Collection.configure>` and `getting |
| 382 |
<.Context.config>` APIs enable moving otherwise 'hardcoded' default values into |
| 383 |
a config structure which downstream users are free to redefine. Let's apply |
| 384 |
this to our example. First we add an explicit namespace object:: |
| 385 |
|
| 386 |
from invoke import Collection, task |
| 387 |
|
| 388 |
default_target = "docs/_build" |
| 389 |
|
| 390 |
@task |
| 391 |
def clean(c, target=default_target): |
| 392 |
c.run("rm -rf {}".format(target)) |
| 393 |
|
| 394 |
@task |
| 395 |
def build(c, target=default_target): |
| 396 |
c.run("sphinx-build docs {}".format(target)) |
| 397 |
|
| 398 |
ns = Collection(clean, build) |
| 399 |
|
| 400 |
Then we can move the default build target value into the collection's default |
| 401 |
configuration, and refer to it via the context. At this point we also change |
| 402 |
our kwarg default value to be ``None`` so we can determine whether or not a |
| 403 |
runtime value was given. The result:: |
| 404 |
|
| 405 |
@task |
| 406 |
def clean(c, target=None): |
| 407 |
if target is None: |
| 408 |
target = c.sphinx.target |
| 409 |
c.run("rm -rf {}".format(target)) |
| 410 |
|
| 411 |
@task |
| 412 |
def build(c, target=None): |
| 413 |
if target is None: |
| 414 |
target = c.sphinx.target |
| 415 |
c.run("sphinx-build docs {}".format(target)) |
| 416 |
|
| 417 |
ns = Collection(clean, build) |
| 418 |
ns.configure({'sphinx': {'target': "docs/_build"}}) |
| 419 |
|
| 420 |
The result isn't significantly more complex than what we began with, and as |
| 421 |
we'll see next, it's now trivial for users to override your defaults in various |
| 422 |
ways. |
| 423 |
|
| 424 |
Configuration overriding |
| 425 |
------------------------ |
| 426 |
|
| 427 |
The lowest-level override is, of course, just modifying the local `.Collection` |
| 428 |
tree into which a distributed module has been imported. E.g. if the above |
| 429 |
module is distributed as ``myproject.docs``, someone can define a ``tasks.py`` |
| 430 |
that does this:: |
| 431 |
|
| 432 |
from invoke import Collection, task |
| 433 |
from myproject import docs |
| 434 |
|
| 435 |
@task |
| 436 |
def mylocaltask(c): |
| 437 |
# Some local stuff goes here |
| 438 |
pass |
| 439 |
|
| 440 |
# Add 'docs' to our local root namespace, plus our own task |
| 441 |
ns = Collection(mylocaltask, docs) |
| 442 |
|
| 443 |
And then they can add this to the bottom:: |
| 444 |
|
| 445 |
# Our docs live in 'built_docs', not 'docs/_build' |
| 446 |
ns.configure({'sphinx': {'target': "built_docs"}}) |
| 447 |
|
| 448 |
Now we have a ``docs`` sub-namespace whose build target defaults to |
| 449 |
``built_docs`` instead of ``docs/_build``. Runtime users can still override |
| 450 |
this via flags (e.g. ``inv docs.build --target='some/other/dir'``) just as |
| 451 |
before. |
| 452 |
|
| 453 |
If you prefer configuration files over in-Python tweaking of your namespace |
| 454 |
tree, that works just as well; instead of adding the line above to the previous |
| 455 |
snippet, instead drop this into a file next to ``tasks.py`` named |
| 456 |
``invoke.yaml``:: |
| 457 |
|
| 458 |
sphinx: |
| 459 |
target: built_docs |
| 460 |
|
| 461 |
For this example, that sort of local-to-project conf file makes the most sense, |
| 462 |
but don't forget that the :ref:`config hierarchy <config-hierarchy>` offers |
| 463 |
additional configuration methods which may be suitable depending on your needs. |
| 464 |
|
| 465 |
|
| 466 |
.. rubric:: Footnotes |
| 467 |
|
| 468 |
.. [1] |
| 469 |
Copying and modifying the file breaks code reuse; overriding the |
| 470 |
module-level ``default_path`` variable won't play well with concurrency; |
| 471 |
wrapping the tasks with different default arguments works but is fragile |
| 472 |
and adds boilerplate. |