Debian Codegraph @

docs/onboarding.rst

raw
1 ###########
2 Onboarder
3 ###########
4
5 This guide walks through tox's internal architecture — how the pieces fit together, what each subsystem does, and where
6 to look when contributing. It is aimed at **first-time contributors** and new team members.
7
8 **How to use this guide:**
9
10 - **Part 1** gives you the big picture and tells you where to start reading code.
11 - **Part 2** follows tox's execution flow from startup to completion.
12 - **Part 3** covers supporting systems you'll need when working on specific features.
13 - **Part 4** contains reference materials for looking up details.
14
15 **********************
16 Part 1: Get Oriented
17 **********************
18
19 Quick facts
20 ===========
21
22 .. list-table::
23 :widths: 25 75
24
25 - - **Language**
26 - `Python <https://www.python.org/>`_ ≥ 3.10
27 - - **Build system**
28 - `hatchling <https://hatch.pypa.io/latest/config/build/>`_ (`PEP 517 <https://peps.python.org/pep-0517/>`_ / `PEP
29 660 <https://peps.python.org/pep-0660/>`_)
30 - - **Plugin framework**
31 - `pluggy <https://pluggy.readthedocs.io/>`_
32 - - **Test framework**
33 - `pytest <https://pytest.org/>`_ (+ `xdist <https://pytest-xdist.readthedocs.io/>`_, `pytest-mock
34 <https://pytest-mock.readthedocs.io/>`_)
35 - - **Docs**
36 - `Sphinx <https://www.sphinx-doc.org/>`_ + `Furo <https://github.com/pradyunsg/furo>`_ theme
37 - - **License**
38 - `MIT <https://opensource.org/licenses/MIT>`_
39
40 Repository layout
41 =================
42
43 ::
44
45 tox/
46 ├── src/tox/ # main package
47 │ ├── run.py # entry point
48 │ ├── provision.py # self-provisioning logic
49 │ ├── report.py # logging / colored output
50 │ ├── config/ # configuration subsystem
51 │ │ ├── cli/ # CLI parser (two-phase)
52 │ │ ├── loader/ # INI / TOML / memory loaders
53 │ │ └── source/ # config file discovery
54 │ ├── session/ # runtime state + subcommands
55 │ │ └── cmd/ # built-in subcommands
56 │ ├── tox_env/ # environment classes
57 │ │ └── python/ # Python-specific envs
58 │ │ └── virtual_env/ # virtualenv-based concrete envs
59 │ ├── execute/ # subprocess execution
60 │ ├── plugin/ # pluggy integration
61 │ ├── journal/ # JSON result journal
62 │ └── util/ # helpers (graph, cpu, path, …)
63 ├── tests/ # mirrors src/tox/ structure
64 └── docs/ # Sphinx documentation
65
66 Start here: Recommended reading order
67 =====================================
68
69 For a first-time contributor, read the code in this order:
70
71 1. `src/tox/run.py <https://github.com/tox-dev/tox/blob/main/src/tox/run.py>`_ — entry point (~50 lines)
72 2. `src/tox/session/state.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/state.py>`_ — state construction
73 3. `src/tox/provision.py <https://github.com/tox-dev/tox/blob/main/src/tox/provision.py>`_ — provisioning logic
74 4. `src/tox/session/env_select.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/env_select.py>`_ — env
75 discovery
76 5. `src/tox/tox_env/api.py <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/api.py>`_ — base environment
77 6. `src/tox/execute/api.py <https://github.com/tox-dev/tox/blob/main/src/tox/execute/api.py>`_ — execution framework
78 7. `src/tox/config/main.py <https://github.com/tox-dev/tox/blob/main/src/tox/config/main.py>`_ — configuration
79
80 The big picture: End-to-end flow
81 ================================
82
83 This traces ``tox run -e py311`` from start to finish, showing the six main phases:
84
85 .. mermaid::
86
87 flowchart LR
88 subgraph ENTRY["1️⃣ ENTRY POINT"]
89 direction TB
90 A[tox run -e py311]
91 B[Parse CLI]
92 C[Load config]
93 D[Build State]
94 A --> B --> C --> D
95 end
96
97 subgraph PROV["2️⃣ PROVISIONING"]
98 direction TB
99 E{Requirements<br/>satisfied?}
100 F[Create provision env]
101 G[Re-invoke tox]
102 E -->|No| F --> G
103 end
104
105 subgraph DISC["3️⃣ DISCOVERY"]
106 direction TB
107 H[Dispatch subcommand]
108 I[Discover envs]
109 J[Resolve dependencies]
110 H --> I --> J
111 end
112
113 subgraph SETUP["4️⃣ ENVIRONMENT SETUP"]
114 direction TB
115 K{Virtualenv<br/>exists?}
116 L[Create virtualenv]
117 M[Install dependencies]
118 N[Build & install package]
119 O[Run extra_setup_commands]
120 K -->|No| L --> M
121 K -->|Yes| M
122 M --> N --> O
123 end
124
125 subgraph EXEC["5️⃣ EXECUTION"]
126 direction TB
127 P[Run commands_pre]
128 Q[Run commands]
129 R[Run commands_post]
130 P --> Q --> R
131 end
132
133 subgraph RESULT["6️⃣ RESULTS"]
134 direction TB
135 S[Collect outcomes]
136 T[Write journal]
137 U[Report summary]
138 S --> T --> U
139 end
140
141 ENTRY --> PROV
142 G -.Re-invoke.-> ENTRY
143 PROV -->|Yes| DISC
144 DISC --> SETUP
145 SETUP --> EXEC
146 EXEC --> RESULT
147
148 style ENTRY fill:#E8F5E9,stroke:#4CAF50,stroke-width:2px
149 style PROV fill:#FFF3E0,stroke:#FF9800,stroke-width:2px
150 style DISC fill:#E3F2FD,stroke:#2196F3,stroke-width:2px
151 style SETUP fill:#F3E5F5,stroke:#9C27B0,stroke-width:2px
152 style EXEC fill:#E1F5FE,stroke:#03A9F4,stroke-width:2px
153 style RESULT fill:#FFEBEE,stroke:#F44336,stroke-width:2px
154
155 style A fill:#5CB85C,stroke:#3D8B40,color:#fff
156 style B fill:#5CB85C,stroke:#3D8B40,color:#fff
157 style C fill:#5CB85C,stroke:#3D8B40,color:#fff
158 style D fill:#5CB85C,stroke:#3D8B40,color:#fff
159 style E fill:#F0AD4E,stroke:#C88A3C,color:#fff
160 style F fill:#F0AD4E,stroke:#C88A3C,color:#fff
161 style G fill:#F0AD4E,stroke:#C88A3C,color:#fff
162 style H fill:#4A90E2,stroke:#2E5C8A,color:#fff
163 style I fill:#4A90E2,stroke:#2E5C8A,color:#fff
164 style J fill:#4A90E2,stroke:#2E5C8A,color:#fff
165 style K fill:#9B59B6,stroke:#6D3F8C,color:#fff
166 style L fill:#9B59B6,stroke:#6D3F8C,color:#fff
167 style M fill:#9B59B6,stroke:#6D3F8C,color:#fff
168 style N fill:#9B59B6,stroke:#6D3F8C,color:#fff
169 style O fill:#9B59B6,stroke:#6D3F8C,color:#fff
170 style P fill:#3498DB,stroke:#2574A9,color:#fff
171 style Q fill:#3498DB,stroke:#2574A9,color:#fff
172 style R fill:#3498DB,stroke:#2574A9,color:#fff
173 style S fill:#E74C3C,stroke:#B23A2F,color:#fff
174 style T fill:#E74C3C,stroke:#B23A2F,color:#fff
175 style U fill:#E74C3C,stroke:#B23A2F,color:#fff
176
177 High-level architecture
178 =======================
179
180 The following diagram shows the main flow through tox's subsystems:
181
182 .. mermaid::
183
184 flowchart TD
185 A[CLI / argv]
186 B[Two-Phase Parser<br/>cli/]
187 C[Config<br/>config/main.py<br/>Sources → Loaders → Types]
188 D[State<br/>session/state.py<br/>owns Config + EnvSelector]
189 E[Provisioning<br/>provision.py<br/>ensures requires/min_ver]
190 F[Subcommand Handler<br/>run / list / config / …]
191 G[EnvSelector<br/>env_select.py<br/>discovers + builds ToxEnv instances]
192 H[RunToxEnv<br/>commands, deps]
193 I[PackageToxEnv<br/>wheel / sdist]
194 J[Execute<br/>subprocess mgmt]
195
196 A --> B --> C --> D --> E --> F --> G
197 G --> H
198 G --> I
199 H --> J
200
201 style A fill:#E74C3C,stroke:#B23A2F,color:#fff
202 style B fill:#5CB85C,stroke:#3D8B40,color:#fff
203 style C fill:#4A90E2,stroke:#2E5C8A,color:#fff
204 style D fill:#9B59B6,stroke:#6D3F8C,color:#fff
205 style E fill:#F0AD4E,stroke:#C88A3C,color:#fff
206 style F fill:#E67E22,stroke:#B85E18,color:#fff
207 style G fill:#3498DB,stroke:#2574A9,color:#fff
208 style H fill:#27AE60,stroke:#1E8449,color:#fff
209 style I fill:#16A085,stroke:#117864,color:#fff
210 style J fill:#C0392B,stroke:#943126,color:#fff
211
212 *****************************
213 Part 2: Main Execution Flow
214 *****************************
215
216 This section follows the chronological execution path through tox, from startup to completion.
217
218 .. _exec-boot-sequence:
219
220 Boot sequence
221 =============
222
223 All roads lead to `tox.run.run() <https://github.com/tox-dev/tox/blob/main/src/tox/run.py>`_:
224
225 .. mermaid::
226
227 flowchart TD
228 A[run]
229 B[main]
230 C[setup_report]
231 D[get_options]
232 E[setup_state]
233 F{provision check?}
234 G[Create provision env]
235 H[handler]
236
237 A --> B --> C --> D --> E --> F
238 F -->|Missing| G
239 F -->|OK| H
240 G -.Re-invoke.-> A
241
242 style A fill:#5CB85C,stroke:#3D8B40,color:#fff
243 style B fill:#5CB85C,stroke:#3D8B40,color:#fff
244 style C fill:#5CB85C,stroke:#3D8B40,color:#fff
245 style D fill:#4A90E2,stroke:#2E5C8A,color:#fff
246 style E fill:#9B59B6,stroke:#6D3F8C,color:#fff
247 style F fill:#F0AD4E,stroke:#C88A3C,color:#fff
248 style G fill:#E67E22,stroke:#B85E18,color:#fff
249 style H fill:#3498DB,stroke:#2574A9,color:#fff
250
251 .. _exec-entry-points:
252
253 Entry points & CLI
254 ==================
255
256 The CLI parsing and command dispatch system handles argument processing, plugin loading, and routing to subcommands.
257
258 Two-phase CLI parsing
259 ---------------------
260
261 Phase 1 — *bootstrap* (before plugins load):
262
263 - It parses only ``-c`` / ``--conf`` (config path) and ``-v`` / ``-q`` (verbosity).
264 - It sets up logging (see :ref:`sys-reporting`).
265 - It discovers configuration source (see :ref:`sys-configuration`).
266 - It loads plugins (see :ref:`sys-plugins`).
267
268 Phase 2 — *full parse* (after plugins):
269
270 - Plugins register their arguments via the ``tox_add_option`` hook (see :ref:`sys-plugins`).
271 - It performs a full ``argparse`` parse over all arguments.
272 - Subcommand handlers are stored in ``Parsed.cmd_handlers``.
273
274 Key classes
275 -----------
276
277 - `ToxParser <https://github.com/tox-dev/tox/blob/main/src/tox/config/cli/parser.py>`_ — subclass of
278 :class:`argparse.ArgumentParser`. Supports ``add_command()`` which creates a sub-parser and registers the handler.
279 - `Parsed <https://github.com/tox-dev/tox/blob/main/src/tox/config/cli/parse.py>`_ — the result of Phase 2. Holds
280 ``options``, ``pos_args``, ``cmd_handlers``, ``env`` (a ``CliEnv``), and ``override`` (from ``-x``).
281 - **Options** — the :class:`argparse.Namespace`. Notably has ``command`` (the chosen subcommand name).
282
283 Built-in subcommands
284 --------------------
285
286 .. list-table::
287 :header-rows: 1
288
289 - - Command
290 - Alias
291 - Module
292 - Purpose
293 - - ``run``
294 - ``r``
295 - `session/cmd/run/sequential.py
296 <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/run/sequential.py>`_
297 - Run environments sequentially (default)
298 - - ``run-parallel``
299 - ``p``
300 - `session/cmd/run/parallel.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/run/parallel.py>`_
301 - Run environments in parallel
302 - - ``list``
303 - ``l``
304 - `session/cmd/list_envs.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/list_envs.py>`_
305 - List available environments
306 - - ``config``
307 - ``c``
308 - `session/cmd/show_config.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/show_config.py>`_
309 - Show materialised configuration
310 - - ``devenv``
311 - ``d``
312 - `session/cmd/devenv.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/devenv.py>`_
313 - Create a development environment
314 - - ``exec``
315 - ``e``
316 - `session/cmd/exec_.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/exec_.py>`_
317 - Run an arbitrary command in an env
318 - - ``depends``
319 - ``de``
320 - `session/cmd/depends.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/depends.py>`_
321 - Visualize environment dependency graph
322 - - ``quickstart``
323 - ``q``
324 - `session/cmd/quickstart.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/quickstart.py>`_
325 - Generate a starter tox config
326 - - ``legacy``
327 - ``le``
328 - `session/cmd/legacy.py <https://github.com/tox-dev/tox/blob/main/src/tox/session/cmd/legacy.py>`_
329 - Backward-compatible tox 3 entry point
330
331 .. _exec-session:
332
333 Session & state management
334 ==========================
335
336 State
337 -----
338
339 `State <https://github.com/tox-dev/tox/blob/main/src/tox/session/state.py>`_ is the top-level runtime container:
340
341 .. list-table::
342 :header-rows: 1
343
344 - - Attribute
345 - Type
346 - Purpose
347 - - ``conf``
348 - `Config <https://github.com/tox-dev/tox/blob/main/src/tox/config/main.py>`_
349 - Fully materialised tox configuration
350 - - ``_options``
351 - `Parsed <https://github.com/tox-dev/tox/blob/main/src/tox/config/cli/parse.py>`_
352 - CLI parse result
353 - - ``_args``
354 - ``list[str]``
355 - Original CLI args (re-invocation during provisioning)
356 - - ``journal``
357 - `Journal <https://github.com/tox-dev/tox/blob/main/src/tox/journal/__init__.py>`_
358 - Run journal for ``--result-json``
359 - - ``envs``
360 - `EnvSelector <https://github.com/tox-dev/tox/blob/main/src/tox/session/env_select.py>`_
361 - Lazily created; single entry point for all env discovery
362
363 The ``envs`` property creates the `EnvSelector
364 <https://github.com/tox-dev/tox/blob/main/src/tox/session/env_select.py>`_ on first access — typically when the
365 subcommand handler starts iterating environments.
366
367 CliEnv
368 ------
369
370 `CliEnv <https://github.com/tox-dev/tox/blob/main/src/tox/config/cli/env.py>`_ captures the user's ``-e`` flag in three
371 modes:
372
373 - **Specific list** (``-e py39,py310``) — run exactly these envs.
374 - **ALL** (``-e ALL``) — run every defined env.
375 - **Default** (no ``-e``) — use ``env_list`` from config.
376
377 EnvSelector
378 -----------
379
380 The core environment discovery engine `EnvSelector
381 <https://github.com/tox-dev/tox/blob/main/src/tox/session/env_select.py>`_:
382
383 1. **Collect names** from config, CLI ``-e``, labels (``-m``), factors (``-f``).
384 2. **Phase 1**: For each name, build a :class:`~tox.tox_env.api.ToxEnv` via ``_build_run_env()``, then query its
385 packaging needs and build package envs.
386 3. **Phase 2**: Reorder to match original definition order, apply label/factor filtering.
387 4. **Iterate**: ``iter()`` yields envs filtered by active status, run-env-only, and ``--skip-env`` regex.
388
389 Subcommand dispatch
390 -------------------
391
392 After provisioning, the handler is dispatched (see :ref:`exec-entry-points` for built-in subcommands):
393
394 .. code-block:: python
395
396 handler = state._options.cmd_handlers[state.conf.options.command]
397 return handler(state)
398
399 Run execution uses ``ready_to_run_envs()`` for topological batch scheduling — environments whose ``depends`` are all
400 satisfied are yielded first.
401
402 .. _exec-environment-lifecycle:
403
404 Environment lifecycle
405 =====================
406
407 .. mermaid::
408
409 flowchart TD
410 A["1. SETUP<br/>• Platform check (skip if no match)<br/>• Recreate check → clean()<br/>• Create virtualenv<br/>• Install deps<br/>• Install package (build + install)<br/>• extra_setup_commands"]
411 B["2. EXECUTE<br/>• commands_pre<br/>• commands<br/>• commands_post (always runs)"]
412 C["3. TEARDOWN<br/>• Detach from package envs<br/>• Close PEP-517 backend<br/>• Delete built packages"]
413
414 A --> B --> C
415
416 style A fill:#5CB85C,stroke:#3D8B40,color:#fff
417 style B fill:#F0AD4E,stroke:#C88A3C,color:#fff
418 style C fill:#E74C3C,stroke:#B23A2F,color:#fff
419
420 If a `Recreate <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/errors.py>`_ exception is raised during setup
421 (e.g., Python version changed), tox automatically cleans and retries.
422
423 .. _exec-execution:
424
425 Execution system
426 ================
427
428 .. mermaid::
429
430 flowchart TD
431 A["Execute (ABC)<br/>execute/api.py"]
432 B["LocalSubProcessExecutor<br/>execute/local_sub_process/"]
433
434 A --> B
435
436 style A fill:#9B59B6,stroke:#6D3F8C,color:#fff
437 style B fill:#4A90E2,stroke:#2E5C8A,color:#fff
438
439 Command flow
440 ------------
441
442 1. :meth:`ToxEnv.execute() <tox.tox_env.api.ToxEnv.execute>` builds an `ExecuteRequest
443 <https://github.com/tox-dev/tox/blob/main/src/tox/execute/request.py>`_.
444 2. `Execute.call() <https://github.com/tox-dev/tox/blob/main/src/tox/execute/api.py>`_ — context manager — creates
445 `SyncWrite <https://github.com/tox-dev/tox/blob/main/src/tox/execute/stream.py>`_ streams for stdout/stderr.
446 3. `ExecuteInstance.start()
447 <https://github.com/tox-dev/tox/blob/main/src/tox/execute/local_sub_process/execute_instance.py>`_ spawns a
448 :class:`subprocess.Popen` subprocess and starts drain threads.
449 4. `ExecuteStatus.wait()
450 <https://github.com/tox-dev/tox/blob/main/src/tox/execute/local_sub_process/execute_status.py>`_ blocks until
451 completion.
452 5. An `Outcome <https://github.com/tox-dev/tox/blob/main/src/tox/execute/request.py>`_ is constructed with exit code,
453 captured output, and timing.
454
455 Command resolution
456 ------------------
457
458 Before execution, tox performs the following steps:
459
460 - It resolves the executable via :func:`shutil.which` using the env's ``PATH``.
461 - It checks against ``allowlist_externals`` glob patterns.
462 - On Unix, it checks for ``TOX_LIMITED_SHEBANG`` and prepends the shebang interpreter.
463
464 Output capture
465 --------------
466
467 The `SyncWrite <https://github.com/tox-dev/tox/blob/main/src/tox/execute/stream.py>`_ class accumulates all bytes in a
468 :class:`io.BytesIO` buffer (thread-safe):
469
470 - If ``show=True``, it forwards to real stdout/stderr in real-time.
471 - It uses a 0.1s timer to flush partial lines.
472 - Stderr gets colored **red** when color is enabled.
473
474 Interrupt mechanism
475 -------------------
476
477 Escalating stop when tox receives a signal:
478
479 1. Wait ``cmd_kill_delay`` (default 0.0s) for voluntary exit.
480 2. Send ``SIGINT`` (Unix) / ``Ctrl+C`` (Windows), wait 0.3s.
481 3. Send ``SIGTERM``, wait 0.2s.
482 4. Send ``SIGKILL`` (Unix only), unconditional wait.
483
484 ****************************
485 Part 3: Supporting Systems
486 ****************************
487
488 These subsystems support the main execution flow. Read these when working on specific features.
489
490 .. _sys-configuration:
491
492 Configuration system
493 ====================
494
495 The configuration system converts raw text (from INI/TOML files, CLI flags, and environment variables) into
496 strongly-typed Python objects.
497
498 Layer diagram
499 -------------
500
501 .. mermaid::
502
503 flowchart TD
504 A[CLI Options<br/>Parsed]
505 B[Source<br/>tox.ini / pyproject.toml / tox.toml / setup.cfg<br/>provides Loaders per section]
506 C[Config<br/>main.py<br/>central registry of all ConfigSets]
507 D[ConfigSet<br/>sets.py<br/>group of named config keys for one env/core]
508 E[ConfigDefinition<br/>of_type.py<br/>single key with type, default, post-process]
509 F[Loader<br/>loader/api.py<br/>reads raw values from the source]
510 G[Convert / StrConvert<br/>loader/convert.py, loader/str_convert.py]
511
512 A --> B
513 B --> C
514 C --> D
515 D --> E
516 E --> F
517 F --> G
518
519 style A fill:#4A90E2,stroke:#2E5C8A,color:#fff
520 style B fill:#5CB85C,stroke:#3D8B40,color:#fff
521 style C fill:#F0AD4E,stroke:#C88A3C,color:#fff
522 style D fill:#9B59B6,stroke:#6D3F8C,color:#fff
523 style E fill:#E67E22,stroke:#B85E18,color:#fff
524 style F fill:#3498DB,stroke:#2574A9,color:#fff
525 style G fill:#E74C3C,stroke:#B23A2F,color:#fff
526
527 Config
528 ------
529
530 The central `Config <https://github.com/tox-dev/tox/blob/main/src/tox/config/main.py>`_ object is created once per
531 session:
532
533 - The ``core`` property lazily creates `CoreConfigSet
534 <https://github.com/tox-dev/tox/blob/main/src/tox/config/sets.py>`_ and attaches loaders.
535 - The ``get_env(name)`` method returns `EnvConfigSet <https://github.com/tox-dev/tox/blob/main/src/tox/config/sets.py>`_
536 for a tox environment and resolves sections and base inheritance.
537 - The ``memory_seed_loaders`` is a ``defaultdict(list[MemoryLoader])`` that injects config values programmatically.
538 - The ``env_names()`` method iterates environment names from source and plugin-extended envs.
539
540 Source discovery
541 ----------------
542
543 The `discover_source() <https://github.com/tox-dev/tox/blob/main/src/tox/config/source/discover.py>`_ function searches
544 for configuration in this order:
545
546 1. If ``--conf`` is specified, it uses that file directly.
547 2. Otherwise, it walks up from CWD looking for:
548
549 - ``tox.toml`` which uses `TomlSource <https://github.com/tox-dev/tox/blob/main/src/tox/config/source/toml_.py>`_.
550 - ``pyproject.toml`` (with ``[tool.tox]``) which uses `PyProjectTomlSource
551 <https://github.com/tox-dev/tox/blob/main/src/tox/config/source/pyproject.py>`_.
552 - ``tox.ini`` which uses `IniSource <https://github.com/tox-dev/tox/blob/main/src/tox/config/source/ini.py>`_.
553 - ``setup.cfg`` (with ``[tox:tox]``) which uses `SetupCfgSource
554 <https://github.com/tox-dev/tox/blob/main/src/tox/config/source/setup_cfg.py>`_.
555
556 3. If nothing is found, it creates an empty `IniSource
557 <https://github.com/tox-dev/tox/blob/main/src/tox/config/source/ini.py>`_ at CWD.
558
559 Source hierarchy:
560
561 .. mermaid::
562
563 flowchart TD
564 A["Source (ABC)<br/>source/api.py"]
565 B["IniSource<br/>source/ini.py"]
566 C["SetupCfgSource<br/>source/setup_cfg.py"]
567 D["TomlSource<br/>source/toml_.py"]
568 E["PyProjectTomlSource<br/>source/pyproject.py"]
569
570 A --> B
571 A --> D
572 B --> C
573 D --> E
574
575 style A fill:#9B59B6,stroke:#6D3F8C,color:#fff
576 style B fill:#4A90E2,stroke:#2E5C8A,color:#fff
577 style C fill:#3498DB,stroke:#2574A9,color:#fff
578 style D fill:#5CB85C,stroke:#3D8B40,color:#fff
579 style E fill:#27AE60,stroke:#1E8449,color:#fff
580
581 Loaders
582 -------
583
584 .. list-table::
585 :header-rows: 1
586
587 - - Loader
588 - Reads from
589 - - `IniLoader <https://github.com/tox-dev/tox/blob/main/src/tox/config/loader/ini.py>`_
590 - INI ``[section]`` key=value pairs
591 - - `TomlLoader <https://github.com/tox-dev/tox/blob/main/src/tox/config/loader/toml.py>`_
592 - TOML ``[table]`` key-value pairs
593 - - `MemoryLoader <https://github.com/tox-dev/tox/blob/main/src/tox/config/loader/memory.py>`_
594 - In-memory dict (provisioning, plugins)
595
596 How a value is resolved
597 -----------------------
598
599 When you access ``env_config["deps"]``:
600
601 1. `ConfigDynamicDefinition.__call__() <https://github.com/tox-dev/tox/blob/main/src/tox/config/of_type.py>`_ is
602 invoked.
603 2. It checks its cache.
604 3. Iterates through the attached **loaders** (INI loader, TOML loader, memory loader).
605 4. The first loader that has the key returns the raw value.
606 5. The raw value is **converted** to the target type (via `Convert
607 <https://github.com/tox-dev/tox/blob/main/src/tox/config/loader/convert.py>`_).
608 6. **Overrides** from ``--override`` are applied.
609 7. ``post_process`` callback runs (if registered).
610 8. The result is cached.
611
612 .. _sys-plugins:
613
614 Plugin system
615 =============
616
617 Tox uses `pluggy <https://pluggy.readthedocs.io/>`_ with the ``"tox"`` namespace.
618
619 Hooks
620 -----
621
622 .. list-table::
623 :header-rows: 1
624
625 - - Hook
626 - When called
627 - - ``tox_register_tox_env``
628 - Register custom run/package env types
629 - - ``tox_extend_env_list``
630 - Declare additional env names dynamically
631 - - ``tox_add_option``
632 - Add CLI arguments (after logging + config discovery)
633 - - ``tox_add_core_config``
634 - Core configuration is being built
635 - - ``tox_add_env_config``
636 - Per-environment configuration is being built
637 - - ``tox_before_run_commands``
638 - Just before commands execute in an env
639 - - ``tox_after_run_commands``
640 - Just after commands execute
641 - - ``tox_on_install``
642 - Before executing an install command
643 - - ``tox_env_teardown``
644 - After an environment is torn down
645
646 Plugin loading sequence
647 -----------------------
648
649 .. mermaid::
650
651 flowchart TD
652 A["setup_report()<br/>configure logging first"]
653 B["import plugin.manager<br/>triggers MANAGER = ToxPluginManager()"]
654 C["MANAGER._setup()"]
655 D["1. Load inline plugin<br/>toxfile.py / ☣.py"]
656 E["2. Load external plugins<br/>entry points"]
657 F["Respect TOX_DISABLED_EXTERNAL_PLUGINS<br/>env var"]
658 G["3. Load internal plugins<br/>16 hardcoded modules"]
659 H["4. check_pending()<br/>validate all hooks implemented"]
660
661 A --> B --> C
662 C --> D
663 C --> E
664 E --> F
665 C --> G
666 C --> H
667
668 style A fill:#5CB85C,stroke:#3D8B40,color:#fff
669 style B fill:#4A90E2,stroke:#2E5C8A,color:#fff
670 style C fill:#9B59B6,stroke:#6D3F8C,color:#fff
671 style D fill:#F0AD4E,stroke:#C88A3C,color:#fff
672 style E fill:#E67E22,stroke:#B85E18,color:#fff
673 style F fill:#D35400,stroke:#A04000,color:#fff
674 style G fill:#3498DB,stroke:#2574A9,color:#fff
675 style H fill:#E74C3C,stroke:#B23A2F,color:#fff
676
677 Inline plugins
678 --------------
679
680 Tox searches the **parent directory** of the config file for ``toxfile.py`` or ``☣.py``. The first match wins. Example:
681
682 .. code-block:: python
683
684 from tox.plugin import impl
685
686
687 @impl
688 def tox_add_option(parser):
689 parser.add_argument("--my-flag", action="store_true")
690
691
692 @impl
693 def tox_add_env_config(env_conf, state):
694 env_conf.add_config("my_key", of_type=str, default="", desc="My custom key")
695
696 External plugins
697 ----------------
698
699 Any installed package declaring a ``tox`` entry point group is discovered:
700
701 .. code-block:: toml
702
703 # In the plugin's pyproject.toml
704 [project.entry-points.tox]
705 my_plugin = "my_tox_plugin"
706
707 Set ``TOX_DISABLED_EXTERNAL_PLUGINS`` to a comma-separated list of plugin names to block specific plugins.
708
709 Packaging details
710 =================
711
712 REGISTER singleton
713 ------------------
714
715 The `REGISTER <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/register.py>`_ singleton maintains two
716 registries:
717
718 - ``_run_envs`` stores runner types keyed by ``id()`` string.
719 - ``_package_envs`` stores packager types keyed by ``id()`` string.
720 - ``default_env_runner`` holds the first registered runner.
721
722 Plugins register new env types via the ``tox_register_tox_env`` hook (see :ref:`sys-plugins`).
723
724 Package environment
725 -------------------
726
727 A :class:`~tox.tox_env.runner.RunToxEnv` determines if packaging is needed via ``package_envs``:
728
729 - It checks ``no_package`` / ``skip_install`` config.
730 - It returns ``(package_env_name, package_tox_env_type)`` tuples.
731
732 The `Pep517VenvPackager <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/package.py>`_ uses
733 `pyproject_api <https://pyproject-api.readthedocs.io/>`_ to communicate with the build backend (setuptools, flit, etc.)
734 as a subprocess. It supports sdist, wheel, editable (PEP-660), and editable-legacy builds.
735
736 The `VenvCmdBuilder <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/package.py>`_ runs user-defined
737 commands to build packages.
738
739 Package envs are **thread-safe**, meaning a single package env can serve multiple runners in parallel, guarded by
740 ``filelock``.
741
742 Pip installer
743 -------------
744
745 The concrete installer (`tox_env/python/pip/ <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/pip/>`_)
746 has the following features:
747
748 - **Incremental installs** compare new vs cached requirements and only install changes.
749 - It supports ``pip_pre``, constraints, and ``use_frozen_constraints``.
750 - The ``{packages}`` placeholder in ``install_command`` is replaced with actual arguments.
751 - See :ref:`exec-execution` for how installation commands are executed.
752
753 .. _sys-reporting:
754
755 Reporting & logging
756 ===================
757
758 Verbosity levels
759 ----------------
760
761 .. list-table::
762 :header-rows: 1
763
764 - - Verbosity
765 - Flags
766 - Log level
767 - Output
768 - - 0
769 - ``-qq``
770 - CRITICAL
771 - Almost nothing
772 - - 1
773 - ``-q``
774 - ERROR
775 - Errors only
776 - - **2**
777 - *(default)*
778 - **WARNING**
779 - Normal output
780 - - 3
781 - ``-v``
782 - INFO
783 - Detailed progress
784 - - 4
785 - ``-vv``
786 - DEBUG
787 - Debug messages + timestamps
788 - - 5
789 - ``-vvv``
790 - NOTSET
791 - Everything
792
793 ToxHandler
794 ----------
795
796 The central logging handler `ToxHandler <https://github.com/tox-dev/tox/blob/main/src/tox/report.py>`_ is added to the
797 root logger and provides:
798
799 - **Thread-local output** where each thread gets its own stdout/stderr via ``_LogThreadLocal``, enabling parallel
800 execution to capture output separately.
801 - **Colored formatters** where Error = red, Warning = cyan, Info = white.
802 - **Per-environment context** where ``with_context(name)`` sets the current env name for the log prefix
803 (magenta-coloured).
804 - **Command-echo pattern** where messages matching ``"%s%s> %s"`` get special color treatment (magenta env name, green
805 ``>``, reset for command text).
806
807 Journal system
808 --------------
809
810 Optional structured JSON output is enabled by ``--result-json <path>``:
811
812 - `Journal <https://github.com/tox-dev/tox/blob/main/src/tox/journal/__init__.py>`_ (session level) collects
813 ``toxversion``, ``platform``, ``host``, and per-environment results.
814 - `EnvJournal <https://github.com/tox-dev/tox/blob/main/src/tox/journal/env.py>`_ (per-environment) records metadata and
815 execution outcomes (command, stdout, stderr, exit code, elapsed time).
816
817 *****************************
818 Part 4: Reference Materials
819 *****************************
820
821 Class hierarchies
822 =================
823
824 ToxEnv hierarchy
825 ----------------
826
827 .. mermaid::
828
829 flowchart TD
830 A["ToxEnv (ABC)<br/>tox_env/api.py"]
831 B["RunToxEnv (ABC)<br/>tox_env/runner.py"]
832 C["PythonRun<br/>Python + RunToxEnv (ABC)<br/>tox_env/python/runner.py"]
833 D["VirtualEnvRunner<br/>VirtualEnv + PythonRun<br/>virtual_env/runner.py<br/>id = virtualenv"]
834 E["PackageToxEnv (ABC)<br/>tox_env/package.py"]
835 F["PythonPackageToxEnv<br/>Python + PackageToxEnv (ABC)"]
836 G["Pep517VirtualEnvPackager<br/>id = virtualenv-pep-517"]
837 H["VirtualEnvCmdBuilder<br/>id = virtualenv-cmd-builder"]
838 I["Python (ABC, mixin)<br/>tox_env/python/api.py"]
839 J["VirtualEnv (ABC, mixin)<br/>virtual_env/api.py"]
840
841 A --> B
842 A --> E
843 A --> I
844 B --> C
845 C --> D
846 E --> F
847 F --> G
848 F --> H
849 I --> J
850
851 style A fill:#9B59B6,stroke:#6D3F8C,color:#fff
852 style B fill:#4A90E2,stroke:#2E5C8A,color:#fff
853 style C fill:#3498DB,stroke:#2574A9,color:#fff
854 style D fill:#27AE60,stroke:#1E8449,color:#fff
855 style E fill:#F0AD4E,stroke:#C88A3C,color:#fff
856 style F fill:#E67E22,stroke:#B85E18,color:#fff
857 style G fill:#5CB85C,stroke:#3D8B40,color:#fff
858 style H fill:#16A085,stroke:#117864,color:#fff
859 style I fill:#E74C3C,stroke:#B23A2F,color:#fff
860 style J fill:#C0392B,stroke:#943126,color:#fff
861
862 The **diamond pattern** means that leaf classes combine a **role** (runner or packager) with an **implementation
863 strategy** (`VirtualEnv <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/virtual_env/api.py>`_). For
864 example, `VirtualEnvRunner <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/virtual_env/runner.py>`_
865 inherits from both `VirtualEnv <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/virtual_env/api.py>`_
866 and `PythonRun <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/runner.py>`_.
867
868 Design patterns
869 ===============
870
871 .. list-table::
872 :header-rows: 1
873
874 - - Pattern
875 - Where
876 - - **Lazy properties**
877 - `State.envs <https://github.com/tox-dev/tox/blob/main/src/tox/session/state.py>`_, `Config.core
878 <https://github.com/tox-dev/tox/blob/main/src/tox/config/main.py>`_, `EnvSelector._defined_envs
879 <https://github.com/tox-dev/tox/blob/main/src/tox/session/env_select.py>`_
880 - - **Two-phase init**
881 - CLI parsing (bootstrap then full), env build (run then package)
882 - - **Plugin hooks at every seam**
883 - Config, env setup, install, commands, teardown
884 - - **Abstract base + mixins**
885 - :class:`~tox.tox_env.api.ToxEnv` hierarchy (diamond MRO)
886 - - **Registry / singleton**
887 - `REGISTER <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/register.py>`_ for env types, `MANAGER
888 <https://github.com/tox-dev/tox/blob/main/src/tox/plugin/manager.py>`_ for plugins
889 - - **Thread-local state**
890 - `ToxHandler <https://github.com/tox-dev/tox/blob/main/src/tox/report.py>`_'s ``_LogThreadLocal`` for per-env
891 output
892 - - **Incremental caching**
893 - `CacheToxInfo <https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/info.py>`_ (``.tox-info.json``), pip
894 installer diff
895
896 Key decision points
897 ===================
898
899 .. list-table::
900 :header-rows: 1
901
902 - - Point
903 - Decision
904 - Outcome
905 - - Provisioning
906 - Are ``requires`` satisfied?
907 - Continue or re-invoke in venv
908 - - Platform check
909 - Does ``platform`` regex match?
910 - Run or skip env
911 - - Recreate
912 - Config changed since last run?
913 - Wipe and rebuild, or reuse
914 - - Package
915 - ``skip_install`` / ``no_package``?
916 - Build and install, or skip
917 - - Command prefix
918 - ``!`` or ``-`` prefix?
919 - Invert or ignore exit code
920 - - Interrupt
921 - Signal received?
922 - Escalate: INT → TERM → KILL
923
924 Testing guide
925 =============
926
927 The ``tests/`` directory **mirrors** ``src/tox/``:
928
929 ::
930
931 tests/
932 ├── conftest.py # Root fixtures
933 ├── config/ # ← mirrors src/tox/config/
934 │ ├── cli/
935 │ ├── loader/
936 │ └── source/
937 ├── execute/ # ← mirrors src/tox/execute/
938 ├── journal/
939 ├── plugin/
940 ├── session/
941 │ └── cmd/ # Subcommand tests
942 ├── tox_env/
943 │ └── python/
944 ├── util/
945 ├── demo_pkg_inline/ # Minimal PEP-517 test package
946 └── demo_pkg_setuptools/ # Setuptools test package
947
948 Key fixtures
949 ------------
950
951 - **tox_project** creates a temporary project with a tox config and runs tox against it. This is the most important
952 integration test fixture and uses the full flow described in Part 1.
953 - **demo_pkg_inline** is a minimal PEP-517 package with a custom build backend.
954 - **demo_pkg_setuptools** is a setuptools-based package for setuptools-specific tests.