Debian Codegraph @

PKG-INFO

raw
1 Metadata-Version: 2.4
2 Name: cwltool
3 Version: 3.2.20260413085819
4 Summary: Common workflow language reference implementation
5 Home-page: https://github.com/common-workflow-language/cwltool
6 Download-URL: https://github.com/common-workflow-language/cwltool
7 Author: Common workflow language working group
8 Author-email: [email protected]
9 Classifier: Development Status :: 5 - Production/Stable
10 Classifier: Environment :: Console
11 Classifier: Intended Audience :: Developers
12 Classifier: Intended Audience :: Science/Research
13 Classifier: Intended Audience :: Healthcare Industry
14 Classifier: License :: OSI Approved :: Apache Software License
15 Classifier: Natural Language :: English
16 Classifier: Operating System :: MacOS :: MacOS X
17 Classifier: Operating System :: POSIX
18 Classifier: Operating System :: POSIX :: Linux
19 Classifier: Programming Language :: Python :: 3
20 Classifier: Programming Language :: Python :: 3.10
21 Classifier: Programming Language :: Python :: 3.11
22 Classifier: Programming Language :: Python :: 3.12
23 Classifier: Programming Language :: Python :: 3.13
24 Classifier: Programming Language :: Python :: 3.14
25 Classifier: Topic :: Scientific/Engineering
26 Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
27 Classifier: Topic :: Scientific/Engineering :: Astronomy
28 Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
29 Classifier: Topic :: Scientific/Engineering :: Information Analysis
30 Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
31 Classifier: Topic :: System :: Distributed Computing
32 Classifier: Topic :: Utilities
33 Requires-Python: >=3.10, <3.15
34 Description-Content-Type: text/x-rst
35 License-File: LICENSE.txt
36 Requires-Dist: requests>=2.6.1
37 Requires-Dist: ruamel.yaml<0.20,>=0.16
38 Requires-Dist: rdflib<7.7.0,>=4.2.2
39 Requires-Dist: schema-salad<9,>=8.9
40 Requires-Dist: prov==1.5.1
41 Requires-Dist: mypy-extensions
42 Requires-Dist: psutil>=5.6.6
43 Requires-Dist: coloredlogs
44 Requires-Dist: pydot>=1.4.1
45 Requires-Dist: argcomplete>=1.12.0
46 Requires-Dist: pyparsing!=3.0.2
47 Requires-Dist: cwl-utils>=0.41
48 Requires-Dist: spython>=0.3.0
49 Requires-Dist: rich-argparse
50 Requires-Dist: pygments>=2.20.0
51 Requires-Dist: typing-extensions>=4.1.0
52 Provides-Extra: deps
53 Requires-Dist: galaxy-tool-util!=23.0.1,!=23.0.2,!=23.0.3,!=23.0.4,!=23.0.5,<26.1,>=22.1.2; extra == "deps"
54 Requires-Dist: galaxy-util<26.1; extra == "deps"
55 Requires-Dist: pillow; extra == "deps"
56 Dynamic: author
57 Dynamic: author-email
58 Dynamic: classifier
59 Dynamic: description
60 Dynamic: description-content-type
61 Dynamic: download-url
62 Dynamic: home-page
63 Dynamic: license-file
64 Dynamic: provides-extra
65 Dynamic: requires-dist
66 Dynamic: requires-python
67 Dynamic: summary
68
69 #############################################################################################
70 ``cwltool``: The reference implementation of the Common Workflow Language standards
71 #############################################################################################
72
73 |Linux Status| |Coverage Status| |Docs Status|
74
75 PyPI: |PyPI Version| |PyPI Downloads Month| |Total PyPI Downloads|
76
77 Conda: |Conda Version| |Conda Installs|
78
79 Debian: |Debian Testing package| |Debian Stable package|
80
81 Quay.io (Docker): |Quay.io Container|
82
83 .. |Linux Status| image:: https://github.com/common-workflow-language/cwltool/actions/workflows/ci-tests.yml/badge.svg?branch=main
84 :target: https://github.com/common-workflow-language/cwltool/actions/workflows/ci-tests.yml
85
86 .. |Debian Stable package| image:: https://badges.debian.net/badges/debian/stable/cwltool/version.svg
87 :target: https://packages.debian.org/stable/cwltool
88
89 .. |Debian Testing package| image:: https://badges.debian.net/badges/debian/testing/cwltool/version.svg
90 :target: https://packages.debian.org/testing/cwltool
91
92 .. |Coverage Status| image:: https://img.shields.io/codecov/c/github/common-workflow-language/cwltool.svg
93 :target: https://codecov.io/gh/common-workflow-language/cwltool
94
95 .. |PyPI Version| image:: https://badge.fury.io/py/cwltool.svg
96 :target: https://badge.fury.io/py/cwltool
97
98 .. |PyPI Downloads Month| image:: https://pepy.tech/badge/cwltool/month
99 :target: https://pepy.tech/project/cwltool
100
101 .. |Total PyPI Downloads| image:: https://static.pepy.tech/personalized-badge/cwltool?period=total&units=international_system&left_color=black&right_color=orange&left_text=Total%20PyPI%20Downloads
102 :target: https://pepy.tech/project/cwltool
103
104 .. |Conda Version| image:: https://anaconda.org/conda-forge/cwltool/badges/version.svg
105 :target: https://anaconda.org/conda-forge/cwltool
106
107 .. |Conda Installs| image:: https://anaconda.org/conda-forge/cwltool/badges/downloads.svg
108 :target: https://anaconda.org/conda-forge/cwltool
109
110 .. |Quay.io Container| image:: https://quay.io/repository/commonwl/cwltool/status
111 :target: https://quay.io/repository/commonwl/cwltool
112
113 .. |Docs Status| image:: https://readthedocs.org/projects/cwltool/badge/?version=latest
114 :target: https://cwltool.readthedocs.io/en/latest/?badge=latest
115 :alt: Documentation Status
116
117 This is the reference implementation of the `Common Workflow Language open
118 standards <https://www.commonwl.org/>`_. It is intended to be feature complete
119 and provide comprehensive validation of CWL
120 files as well as provide other tools related to working with CWL.
121
122 ``cwltool`` is written and tested for
123 `Python <https://www.python.org/>`_ ``3.x {x = 9, 10, 11, 12, 13}``
124
125 The reference implementation consists of two packages. The ``cwltool`` package
126 is the primary Python module containing the reference implementation in the
127 ``cwltool`` module and console executable by the same name.
128
129 The ``cwlref-runner`` package is optional and provides an additional entry point
130 under the alias ``cwl-runner``, which is the implementation-agnostic name for the
131 default CWL interpreter installed on a host.
132
133 ``cwltool`` is provided by the CWL project, `a member project of Software Freedom Conservancy <https://sfconservancy.org/news/2018/apr/11/cwl-new-member-project/>`_
134 and our `many contributors <https://github.com/common-workflow-language/cwltool/graphs/contributors>`_.
135
136 .. contents:: Table of Contents
137
138 *******
139 Install
140 *******
141
142 ``cwltool`` packages
143 ====================
144
145 Your operating system may offer cwltool directly. For `Debian <https://tracker.debian.org/pkg/cwltool>`_, `Ubuntu <https://launchpad.net/ubuntu/+source/cwltool>`_,
146 and similar Linux distribution try
147
148 .. code:: bash
149
150 sudo apt-get install cwltool
151
152 If you encounter an error, first try to update package information by using
153
154 .. code:: bash
155
156 sudo apt-get update
157
158 If you are running macOS X or other UNIXes and you want to use packages prepared by the conda-forge project, then
159 please follow the install instructions for `conda-forge <https://conda-forge.org/#about>`_ (if you haven't already) and then
160
161 .. code:: bash
162
163 conda install -c conda-forge cwltool
164
165 All of the above methods of installing ``cwltool`` use packages that might contain bugs already fixed in newer versions or be missing desired features.
166 If the packaged version of ``cwltool`` available to you is too old, then we recommend installing using ``pip`` and ``venv``
167
168 .. code:: bash
169
170 python3 -m venv env # Create a virtual environment named 'env' in the current directory
171 source env/bin/activate # Activate environment before installing `cwltool`
172
173 Then install the latest ``cwlref-runner`` package from PyPi (which will install the latest ``cwltool`` package as
174 well)
175
176 .. code:: bash
177
178 pip install cwlref-runner
179
180 If installing alongside another CWL implementation (like ``toil-cwl-runner`` or ``arvados-cwl-runner``) then instead run
181
182 .. code:: bash
183
184 pip install cwltool
185
186 MS Windows users
187 ================
188
189 1. `Install Windows Subsystem for Linux 2 and Docker Desktop <https://docs.docker.com/docker-for-windows/wsl/#prerequisites>`_.
190 2. `Install Debian from the Microsoft Store <https://www.microsoft.com/en-us/p/debian/9msvkqc78pk6>`_.
191 3. Set Debian as your default WSL 2 distro: ``wsl --set-default debian``.
192 4. Return to the Docker Desktop, choose ``Settings`` → ``Resources`` → ``WSL Integration`` and under "Enable integration with additional distros" select "Debian",
193 5. Reboot if you have not yet already.
194 6. Launch Debian and follow the Linux instructions above (``apt-get install cwltool`` or use the ``venv`` method)
195
196 Network problems from within WSL2? Try `these instructions <https://github.com/microsoft/WSL/issues/4731#issuecomment-702176954>`_ followed by ``wsl --shutdown``.
197
198 ``cwltool`` development version
199 ===============================
200
201 Or you can skip the direct ``pip`` commands above and install the latest development version of ``cwltool``:
202
203 .. code:: bash
204
205 git clone https://github.com/common-workflow-language/cwltool.git # clone (copy) the cwltool git repository
206 cd cwltool # Change to source directory that git clone just downloaded
207 pip install .[deps] # Installs ``cwltool`` from source
208 cwltool --version # Check if the installation works correctly
209
210 Remember, if co-installing multiple CWL implementations, then you need to
211 maintain which implementation ``cwl-runner`` points to via a symbolic file
212 system link or `another facility <https://wiki.debian.org/DebianAlternatives>`_.
213
214 Recommended Software
215 ====================
216
217 We strongly suggested to have the following installed:
218
219 * One of the following software container engines
220
221 * `Podman <https://podman.io/getting-started/installation>`_
222 * `Docker <https://docs.docker.com/engine/install/>`_
223 * Singularity/Apptainer: See `Using Singularity`_
224 * udocker: See `Using uDocker`_
225
226 * `node.js <https://nodejs.org/en/download/>`_ for evaluating CWL Expressions quickly
227 (required for `udocker` users, optional but recommended for the other container engines).
228
229 Without these, some examples in the CWL tutorials at http://www.commonwl.org/user_guide/ may not work.
230
231 ***********************
232 Run on the command line
233 ***********************
234
235 Simple command::
236
237 cwl-runner my_workflow.cwl my_inputs.yaml
238
239 Or if you have multiple CWL implementations installed and you want to override
240 the default cwl-runner then use::
241
242 cwltool my_workflow.cwl my_inputs.yml
243
244 You can set cwltool options in the environment with ``CWLTOOL_OPTIONS``,
245 these will be inserted at the beginning of the command line::
246
247 export CWLTOOL_OPTIONS="--debug"
248
249 Use with boot2docker on macOS
250 =============================
251 boot2docker runs Docker inside a virtual machine, and it only mounts ``Users``
252 on it. The default behavior of CWL is to create temporary directories under e.g.
253 ``/Var`` which is not accessible to Docker containers.
254
255 To run CWL successfully with boot2docker you need to set the ``--tmpdir-prefix``
256 and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::
257
258 $ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json
259
260
261 Docker default platform on macOS with Apple Silicon
262 ===================================================
263
264 If macOS users want to run CWL tools/workflows using ``cwltool`` with Docker and their software containers only have support for amd64 (64-bit x86) CPUs, but they have an Apple Silicon (aarch64/arm64) CPU,
265 they run into the error:
266
267 WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested.
268
269 To fix this, export the following environment variable before executing `cwltool`:
270
271 ``export DOCKER_DEFAULT_PLATFORM=linux/amd64``
272
273 To automatically have this variable set in the future, add the same command to ones respective shell profile (e.g. ``~/.zshrc``, ``~/.bash_profile``).
274
275
276 Using uDocker
277 =============
278
279 Some shared computing environments don't support Docker software containers for technical or policy reasons.
280 As a workaround, the CWL reference runner supports using the `udocker <https://github.com/indigo-dc/udocker>`_
281 program on Linux using ``--udocker``.
282
283 udocker installation: https://indigo-dc.github.io/udocker/installation_manual.html
284
285 Run `cwltool` just as you usually would, but with ``--udocker`` prior to the workflow path:
286
287 .. code:: bash
288
289 cwltool --udocker https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/empty.json
290
291 As was mentioned in the `Recommended Software`_ section,
292
293 Using Singularity
294 =================
295
296 ``cwltool`` can also use `Singularity <https://github.com/hpcng/singularity/releases/>`_ version 2.6.1
297 or later as a Docker container runtime.
298 ``cwltool`` with Singularity will run software containers specified in
299 ``DockerRequirement`` and therefore works with Docker images only, native
300 Singularity images are not supported. To use Singularity as the Docker container
301 runtime, provide ``--singularity`` command line option to ``cwltool``.
302 With Singularity, ``cwltool`` can pass all CWL v1.0 conformance tests, except
303 those involving Docker container ENTRYPOINTs.
304
305 Example
306
307 .. code:: bash
308
309 cwltool --singularity https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/cat3-tool-mediumcut.cwl https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/cat-job.json
310
311 Running a tool or workflow from remote or local locations
312 =========================================================
313
314 ``cwltool`` can run tool and workflow descriptions on both local and remote
315 systems via its support for HTTP[S] URLs.
316
317 Input job files and Workflow steps (via the `run` directive) can reference CWL
318 documents using absolute or relative local filesystem paths. If a relative path
319 is referenced and that document isn't found in the current directory, then the
320 following locations will be searched:
321 http://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem
322
323 You can also use `cwldep <https://github.com/common-workflow-language/cwldep>`_
324 to manage dependencies on external tools and workflows.
325
326 Overriding workflow requirements at load time
327 =============================================
328
329 Sometimes a workflow needs additional requirements to run in a particular
330 environment or with a particular dataset. To avoid the need to modify the
331 underlying workflow, cwltool supports requirement "overrides".
332
333 The format of the "overrides" object is a mapping of item identifier (workflow,
334 workflow step, or command line tool) to the process requirements that should be applied.
335
336 .. code:: yaml
337
338 cwltool:overrides:
339 echo.cwl:
340 requirements:
341 EnvVarRequirement:
342 envDef:
343 MESSAGE: override_value
344
345 Overrides can be specified either on the command line, or as part of the job
346 input document. Workflow steps are identified using the name of the workflow
347 file followed by the step name as a document fragment identifier "#id".
348 Override identifiers are relative to the top-level workflow document.
349
350 .. code:: bash
351
352 cwltool --overrides overrides.yml my-tool.cwl my-job.yml
353
354 .. code:: yaml
355
356 input_parameter1: value1
357 input_parameter2: value2
358 cwltool:overrides:
359 workflow.cwl#step1:
360 requirements:
361 EnvVarRequirement:
362 envDef:
363 MESSAGE: override_value
364
365 .. code:: bash
366
367 cwltool my-tool.cwl my-job-with-overrides.yml
368
369
370 Combining parts of a workflow into a single document
371 ====================================================
372
373 Use ``--pack`` to combine a workflow made up of multiple files into a
374 single compound document. This operation takes all the CWL files
375 referenced by a workflow and builds a new CWL document with all
376 Process objects (CommandLineTool and Workflow) in a list in the
377 ``$graph`` field. Cross references (such as ``run:`` and ``source:``
378 fields) are updated to internal references within the new packed
379 document. The top-level workflow is named ``#main``.
380
381 .. code:: bash
382
383 cwltool --pack my-wf.cwl > my-packed-wf.cwl
384
385
386 Running only part of a workflow
387 ===============================
388
389 You can run a partial workflow with the ``--target`` (``-t``) option. This
390 takes the name of an output parameter, workflow step, or input
391 parameter in the top-level workflow. You may provide multiple
392 targets.
393
394 .. code:: bash
395
396 cwltool --target step3 my-wf.cwl
397
398 If a target is an output parameter, it will only run only the steps
399 that contribute to that output. If a target is a workflow step, it
400 will run the workflow starting from that step. If a target is an
401 input parameter, it will only run the steps connected to
402 that input.
403
404 Use ``--print-targets`` to get a listing of the targets of a workflow.
405 To see which steps will run, use ``--print-subgraph`` with
406 ``--target`` to get a printout of the workflow subgraph for the
407 selected targets.
408
409 .. code:: bash
410
411 cwltool --print-targets my-wf.cwl
412
413 cwltool --target step3 --print-subgraph my-wf.cwl > my-wf-starting-from-step3.cwl
414
415
416 Visualizing a CWL document
417 ==========================
418
419 The ``--print-dot`` option will print a file suitable for Graphviz ``dot`` program. Here is a bash onliner to generate a Scalable Vector Graphic (SVG) file:
420
421 .. code:: bash
422
423 cwltool --print-dot my-wf.cwl | dot -Tsvg > my-wf.svg
424
425 Modeling a CWL document as RDF
426 ==============================
427
428 CWL documents can be expressed as RDF triple graphs.
429
430 .. code:: bash
431
432 cwltool --print-rdf --rdf-serializer=turtle mywf.cwl
433
434
435 Environment Variables in cwltool
436 ================================
437
438 This reference implementation supports several ways of setting
439 environment variables for tools, in addition to the standard
440 ``EnvVarRequirement``. The sequence of steps applied to create the
441 environment is:
442
443 0. If the ``--preserve-entire-environment`` flag is present, then begin with the current
444 environment, else begin with an empty environment.
445
446 1. Add any variables specified by ``--preserve-environment`` option(s).
447
448 2. Set ``TMPDIR`` and ``HOME`` per `the CWL v1.0+ CommandLineTool specification <https://www.commonwl.org/v1.0/CommandLineTool.html#Runtime_environment>`_.
449
450 3. Apply any ``EnvVarRequirement`` from the ``CommandLineTool`` description.
451
452 4. Apply any manipulations required by any ``cwltool:MPIRequirement`` extensions.
453
454 5. Substitute any secrets required by ``Secrets`` extension.
455
456 6. Modify the environment in response to ``SoftwareRequirement`` (see below).
457
458
459 Leveraging SoftwareRequirements (Beta)
460 --------------------------------------
461
462 CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool
463 may in turn use to resolve to packages in various package managers or
464 dependency management systems such as `Environment Modules
465 <http://modules.sourceforge.net/>`__.
466
467 Utilizing ``SoftwareRequirement`` hints using cwltool requires an optional
468 dependency, for this reason be sure to use specify the ``deps`` modifier when
469 installing cwltool. For instance::
470
471 $ pip install 'cwltool[deps]'
472
473 Installing cwltool in this fashion enables several new command line options.
474 The most general of these options is ``--beta-dependency-resolvers-configuration``.
475 This option allows one to specify a dependency resolver's configuration file.
476 This file may be specified as either XML or YAML and very simply describes various
477 plugins to enable to "resolve" ``SoftwareRequirement`` dependencies.
478
479 Using these hints will allow cwltool to modify the environment in
480 which your tool runs, for example by loading one or more Environment
481 Modules. The environment is constructed as above, then the environment
482 may modified by the selected tool resolver. This currently means that
483 you cannot override any environment variables set by the selected tool
484 resolver. Note that the environment given to the configured dependency
485 resolver has the variable `_CWLTOOL` set to `1` to allow introspection.
486
487 To discuss some of these plugins and how to configure them, first consider the
488 following ``hint`` definition for an example CWL tool.
489
490 .. code:: yaml
491
492 SoftwareRequirement:
493 packages:
494 - package: seqtk
495 version:
496 - r93
497
498 Now imagine deploying cwltool on a cluster with Software Modules installed
499 and that a ``seqtk`` module is available at version ``r93``. This means cluster
500 users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after
501 sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is
502 available on the ``PATH``. A simple dependency resolvers configuration file, called
503 ``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source
504 the correct module environment before executing the above tool would simply be:
505
506 .. code:: yaml
507
508 - type: modules
509
510 The outer list indicates that one plugin is being enabled, the plugin parameters are
511 defined as a dictionary for this one list item. There is only one required parameter
512 for the plugin above, this is ``type`` and defines the plugin type. This parameter
513 is required for all plugins. The available plugins and the parameters
514 available for each are documented (incompletely) `here
515 <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__.
516 Unfortunately, this documentation is in the context of Galaxy tool
517 ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
518
519 cwltool is distributed with an example of such seqtk tool and sample corresponding
520 job. It could executed from the cwltool root using a dependency resolvers
521 configuration file such as the above one using the command::
522
523 cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \
524 tests/seqtk_seq.cwl \
525 tests/seqtk_seq_job.json
526
527 This example demonstrates both that cwltool can leverage
528 existing software installations and also handle workflows with dependencies
529 on different versions of the same software and libraries. However the above
530 example does require an existing module setup so it is impossible to test this example
531 "out of the box" with cwltool. For a more isolated test that demonstrates all
532 the same concepts - the resolver plugin type ``galaxy_packages`` can be used.
533
534 "Galaxy packages" are a lighter-weight alternative to Environment Modules that are
535 really just defined by a way to lay out directories into packages and versions
536 to find little scripts that are sourced to modify the environment. They have
537 been used for years in Galaxy community to adapt Galaxy tools to cluster
538 environments but require neither knowledge of Galaxy nor any special tools to
539 setup. These should work just fine for CWL tools.
540
541 The cwltool source code repository's test directory is setup with a very simple
542 directory that defines a set of "Galaxy packages" (but really just defines one
543 package named ``random-lines``). The directory layout is simply::
544
545 tests/test_deps_env/
546 random-lines/
547 1.0/
548 env.sh
549
550 If the ``galaxy_packages`` plugin is enabled and pointed at the
551 ``tests/test_deps_env`` directory in cwltool's root and a ``SoftwareRequirement``
552 such as the following is encountered.
553
554 .. code:: yaml
555
556 hints:
557 SoftwareRequirement:
558 packages:
559 - package: 'random-lines'
560 version:
561 - '1.0'
562
563 Then cwltool will simply find that ``env.sh`` file and source it before executing
564 the corresponding tool. That ``env.sh`` script is only responsible for modifying
565 the job's ``PATH`` to add the required binaries.
566
567 This is a full example that works since resolving "Galaxy packages" has no
568 external requirements. Try it out by executing the following command from cwltool's
569 root directory::
570
571 cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf.yml \
572 tests/random_lines.cwl \
573 tests/random_lines_job.json
574
575 The resolvers configuration file in the above example was simply:
576
577 .. code:: yaml
578
579 - type: galaxy_packages
580 base_path: ./tests/test_deps_env
581
582 It is possible that the ``SoftwareRequirement`` s in a given CWL tool will not
583 match the module names for a given cluster. Such requirements can be re-mapped
584 to specific deployed packages or versions using another file specified using
585 the resolver plugin parameter `mapping_files`. We will
586 demonstrate this using `galaxy_packages,` but the concepts apply equally well
587 to Environment Modules or Conda packages (described below), for instance.
588
589 So consider the resolvers configuration file.
590 (`tests/test_deps_env_resolvers_conf_rewrite.yml`):
591
592 .. code:: yaml
593
594 - type: galaxy_packages
595 base_path: ./tests/test_deps_env
596 mapping_files: ./tests/test_deps_mapping.yml
597
598 And the corresponding mapping configuration file (`tests/test_deps_mapping.yml`):
599
600 .. code:: yaml
601
602 - from:
603 name: randomLines
604 version: 1.0.0-rc1
605 to:
606 name: random-lines
607 version: '1.0'
608
609 This is saying if cwltool encounters a requirement of ``randomLines`` at version
610 ``1.0.0-rc1`` in a tool, to rewrite to our specific plugin as ``random-lines`` at
611 version ``1.0``. cwltool has such a test tool called ``random_lines_mapping.cwl``
612 that contains such a source ``SoftwareRequirement``. To try out this example with
613 mapping, execute the following command from the cwltool root directory::
614
615 cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf_rewrite.yml \
616 tests/random_lines_mapping.cwl \
617 tests/random_lines_job.json
618
619 The previous examples demonstrated leveraging existing infrastructure to
620 provide requirements for CWL tools. If instead a real package manager is used
621 cwltool has the opportunity to install requirements as needed. While initial
622 support for Homebrew/Linuxbrew plugins is available, the most developed such
623 plugin is for the `Conda <https://conda.io/docs/#>`__ package manager. Conda has the nice properties
624 of allowing multiple versions of a package to be installed simultaneously,
625 not requiring evaluated permissions to install Conda itself or packages using
626 Conda, and being cross-platform. For these reasons, cwltool may run as a normal
627 user, install its own Conda environment and manage multiple versions of Conda packages
628 on Linux and Mac OS X.
629
630 The Conda plugin can be endlessly configured, but a sensible set of defaults
631 that has proven a powerful stack for dependency management within the Galaxy tool
632 development ecosystem can be enabled by simply passing cwltool the
633 ``--beta-conda-dependencies`` flag.
634
635 With this, we can use the seqtk example above without Docker or any externally managed services - cwltool should install everything it needs
636 and create an environment for the tool. Try it out with the following command::
637
638 cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json
639
640 The CWL specification allows URIs to be attached to ``SoftwareRequirement`` s
641 that allow disambiguation of package names. If the mapping files described above
642 allow deployers to adapt tools to their infrastructure, this mechanism allows
643 tools to adapt their requirements to multiple package managers. To demonstrate
644 this within the context of the seqtk, we can simply break the package name we
645 use and then specify a specific Conda package as follows:
646
647 .. code:: yaml
648
649 hints:
650 SoftwareRequirement:
651 packages:
652 - package: seqtk_seq
653 version:
654 - '1.2'
655 specs:
656 - https://anaconda.org/bioconda/seqtk
657 - https://packages.debian.org/sid/seqtk
658
659 The example can be executed using the command::
660
661 cwltool --beta-conda-dependencies tests/seqtk_seq_wrong_name.cwl tests/seqtk_seq_job.json
662
663 The plugin framework for managing the resolution of these software requirements
664 as maintained as part of `galaxy-tool-util <https://github.com/galaxyproject/galaxy/tree/dev/packages/tool_util>`__ - a small,
665 portable subset of the Galaxy project. More information on configuration and implementation can be found
666 at the following links:
667
668 - `Dependency Resolvers in Galaxy <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__
669 - `Conda for [Galaxy] Tool Dependencies <https://docs.galaxyproject.org/en/latest/admin/conda_faq.html>`__
670 - `Mapping Files - Implementation <https://github.com/galaxyproject/galaxy/commit/495802d229967771df5b64a2f79b88a0eaf00edb>`__
671 - `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
672 - `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
673
674 Use with GA4GH Tool Registry API
675 ================================
676
677 Cwltool can launch tools directly from `GA4GH Tool Registry API`_ endpoints.
678
679 By default, cwltool searches https://dockstore.org/ . Use ``--add-tool-registry`` to add other registries to the search path.
680
681 For example ::
682
683 cwltool quay.io/collaboratory/dockstore-tool-bamstats:develop test.json
684
685 and (defaults to latest when a version is not specified) ::
686
687 cwltool quay.io/collaboratory/dockstore-tool-bamstats test.json
688
689 For this example, grab the test.json (and input file) from https://github.com/CancerCollaboratory/dockstore-tool-bamstats ::
690
691 wget https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-bamstats/versions/develop/PLAIN-CWL/descriptor/test.json
692 wget https://github.com/CancerCollaboratory/dockstore-tool-bamstats/raw/develop/rna.SRR948778.bam
693
694
695 .. _`GA4GH Tool Registry API`: https://github.com/ga4gh/tool-registry-schemas
696
697 Running MPI-based tools that need to be launched
698 ================================================
699
700 Cwltool supports an extension to the CWL spec
701 ``http://commonwl.org/cwltool#MPIRequirement``. When the tool
702 definition has this in its ``requirements``/``hints`` section, and
703 cwltool has been run with ``--enable-ext``, then the tool's command
704 line will be extended with the commands needed to launch it with
705 ``mpirun`` or similar. You can specify the number of processes to
706 start as either a literal integer or an expression (that will result
707 in an integer). For example::
708
709 #!/usr/bin/env cwl-runner
710 cwlVersion: v1.1
711 class: CommandLineTool
712 $namespaces:
713 cwltool: "http://commonwl.org/cwltool#"
714 requirements:
715 cwltool:MPIRequirement:
716 processes: $(inputs.nproc)
717 inputs:
718 nproc:
719 type: int
720
721 Interaction with containers: the MPIRequirement currently prepends its
722 commands to the front of the command line that is constructed. If you
723 wish to run a containerized application in parallel, for simple use
724 cases, this does work with Singularity, depending upon the platform
725 setup. However, this combination should be considered "alpha" -- please
726 do report any issues you have! This does not work with Docker at the
727 moment. (More precisely, you get `n` copies of the same single process
728 image run at the same time that cannot communicate with each other.)
729
730 The host-specific parameters are configured in a simple YAML file
731 (specified with the ``--mpi-config-file`` flag). The allowed keys are
732 given in the following table; all are optional.
733
734 +----------------+------------------+----------+------------------------------+
735 | Key | Type | Default | Description |
736 +================+==================+==========+==============================+
737 | runner | str | "mpirun" | The primary command to use. |
738 +----------------+------------------+----------+------------------------------+
739 | nproc_flag | str | "-n" | Flag to set number of |
740 | | | | processes to start. |
741 +----------------+------------------+----------+------------------------------+
742 | default_nproc | int | 1 | Default number of processes. |
743 +----------------+------------------+----------+------------------------------+
744 | extra_flags | List[str] | [] | A list of any other flags to |
745 | | | | be added to the runner's |
746 | | | | command line before |
747 | | | | the ``baseCommand``. |
748 +----------------+------------------+----------+------------------------------+
749 | env_pass | List[str] | [] | A list of environment |
750 | | | | variables that should be |
751 | | | | passed from the host |
752 | | | | environment through to the |
753 | | | | tool (e.g., giving the |
754 | | | | node list as set by your |
755 | | | | scheduler). |
756 +----------------+------------------+----------+------------------------------+
757 | env_pass_regex | List[str] | [] | A list of python regular |
758 | | | | expressions that will be |
759 | | | | matched against the host's |
760 | | | | environment. Those that match|
761 | | | | will be passed through. |
762 +----------------+------------------+----------+------------------------------+
763 | env_set | Mapping[str,str] | {} | A dictionary whose keys are |
764 | | | | the environment variables set|
765 | | | | and the values being the |
766 | | | | values. |
767 +----------------+------------------+----------+------------------------------+
768
769
770 Enabling Fast Parser (experimental)
771 ===================================
772
773 For very large workflows, `cwltool` can spend a lot of time in
774 initialization, before the first step runs. There is an experimental
775 flag ``--fast-parser`` which can dramatically reduce the
776 initialization overhead, however as of this writing it has several limitations:
777
778 - Error reporting in general is worse than the standard parser, you will want to use it with workflows that you know are already correct.
779
780 - It does not check for dangling links (these will become runtime errors instead of loading errors)
781
782 - Several other cases fail, as documented in https://github.com/common-workflow-language/cwltool/pull/1720
783
784 ***********
785 Development
786 ***********
787
788 Running tests locally
789 =====================
790
791 - Running basic tests ``(/tests)``:
792
793 To run the basic tests after installing `cwltool` execute the following:
794
795 .. code:: bash
796
797 pip install -rtest-requirements.txt
798 pytest ## N.B. This requires node.js or docker to be available
799
800 To run various tests in all supported Python environments, we use `tox <https://github.com/common-workflow-language/cwltool/tree/main/tox.ini>`_. To run the test suite in all supported Python environments
801 first clone the complete code repository (see the ``git clone`` instructions above) and then run
802 the following in the terminal:
803 ``pip install tox; tox -p``
804
805 List of all environment can be seen using:
806 ``tox --listenvs``
807 and running a specific test env using:
808 ``tox run -e <env name>``
809 and additionally run a specific test using this format:
810 ``tox run -e py310-unit -- -v tests/test_examples.py::test_scandeps``
811
812 - Running the entire suite of CWL conformance tests:
813
814 The GitHub repository for the CWL specifications contains a script that tests a CWL
815 implementation against a wide array of valid CWL files using the `cwltest <https://github.com/common-workflow-language/cwltest>`_
816 program
817
818 Instructions for running these tests can be found in the Common Workflow Language Specification repository at https://github.com/common-workflow-language/common-workflow-language/blob/main/CONFORMANCE_TESTS.md .
819
820 Import as a module
821 ==================
822
823 Add
824
825 .. code:: python
826
827 import cwltool
828
829 to your script.
830
831 The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory
832
833 .. code:: python
834
835 import cwltool.factory
836 fac = cwltool.factory.Factory()
837
838 echo = fac.make("echo.cwl")
839 result = echo(inp="foo")
840
841 # result["out"] == "foo"
842
843
844 CWL Tool Control Flow
845 =====================
846
847 Technical outline of how cwltool works internally, for maintainers.
848
849 #. Use CWL ``load_tool()`` to load document.
850
851 #. Fetches the document from file or URL
852 #. Applies preprocessing (syntax/identifier expansion and normalization)
853 #. Validates the document based on cwlVersion
854 #. If necessary, updates the document to the latest spec
855 #. Constructs a Process object using ``make_tool()``` callback. This yields a
856 CommandLineTool, Workflow, or ExpressionTool. For workflows, this
857 recursively constructs each workflow step.
858 #. To construct custom types for CommandLineTool, Workflow, or
859 ExpressionTool, provide a custom ``make_tool()``
860
861 #. Iterate on the ``job()`` method of the Process object to get back runnable jobs.
862
863 #. ``job()`` is a generator method (uses the Python iterator protocol)
864 #. Each time the ``job()`` method is invoked in an iteration, it returns one
865 of: a runnable item (an object with a ``run()`` method), ``None`` (indicating
866 there is currently no work ready to run) or end of iteration (indicating
867 the process is complete.)
868 #. Invoke the runnable item by calling ``run()``. This runs the tool and gets output.
869 #. An output callback reports the output of a process.
870 #. ``job()`` may be iterated over multiple times. It will yield all the work
871 that is currently ready to run and then yield None.
872
873 #. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation.
874
875 #. The WorkflowJob iterates over each WorkflowJobStep and determines if the
876 inputs the step are ready.
877 #. When a step is ready, it constructs an input object for that step and
878 iterates on the ``job()`` method of the workflow job step.
879 #. Each runnable item is yielded back up to top-level run loop
880 #. When a step job completes and receives an output callback, the
881 job outputs are assigned to the output of the workflow step.
882 #. When all steps are complete, the intermediate files are moved to a final
883 workflow output, intermediate directories are deleted, and the workflow's output callback is called.
884
885 #. ``CommandLineTool`` job() objects yield a single runnable object.
886
887 #. The CommandLineTool ``job()`` method calls ``make_job_runner()`` to create a
888 ``CommandLineJob`` object
889 #. The job method configures the CommandLineJob object by setting public
890 attributes
891 #. The job method iterates over file and directories inputs to the
892 CommandLineTool and creates a "path map".
893 #. Files are mapped from their "resolved" location to a "target" path where
894 they will appear at tool invocation (for example, a location inside a
895 Docker container.) The target paths are used on the command line.
896 #. Files are staged to targets paths using either Docker volume binds (when
897 using containers) or symlinks (if not). This staging step enables files
898 to be logically rearranged or renamed independent of their source layout.
899 #. The ``run()`` method of CommandLineJob executes the command line tool or
900 Docker container, waits for it to complete, collects output, and makes
901 the output callback.
902
903 Extension points
904 ================
905
906 The following functions can be passed to main() to override or augment
907 the listed behaviors.
908
909 executor
910 ::
911
912 executor(tool, job_order_object, runtimeContext, logger)
913 (Process, Dict[str, Any], RuntimeContext) -> Tuple[Dict[str, Any], str]
914
915 An implementation of the top-level workflow execution loop should
916 synchronously run a process object to completion and return the
917 output object.
918
919 versionfunc
920 ::
921
922 ()
923 () -> str
924
925 Return version string.
926
927 logger_handler
928 ::
929
930 logger_handler
931 logging.Handler
932
933 Handler object for logging.
934
935 The following functions can be set in LoadingContext to override or
936 augment the listed behaviors.
937
938 fetcher_constructor
939 ::
940
941 fetcher_constructor(cache, session)
942 (Dict[unicode, unicode], requests.sessions.Session) -> Fetcher
943
944 Construct a Fetcher object with the supplied cache and HTTP session.
945
946 resolver
947 ::
948
949 resolver(document_loader, document)
950 (Loader, str | dict[str, Any]) -> str
951
952 Resolve a relative document identifier to an absolute one that can be fetched.
953
954 The following functions can be set in RuntimeContext to override or
955 augment the listed behaviors.
956
957 construct_tool_object
958 ::
959
960 construct_tool_object(toolpath_object, loadingContext)
961 (MutableMapping[str, Any], LoadingContext) -> Process
962
963 Hook to construct a Process object (eg CommandLineTool) object from a document.
964
965 select_resources
966 ::
967
968 selectResources(request)
969 (Dict[str, int], RuntimeContext) -> Dict[str, int]
970
971 Take a resource request and turn it into a concrete resource assignment.
972
973 make_fs_access
974 ::
975
976 make_fs_access(basedir)
977 (str) -> StdFsAccess
978
979 Return a file system access object.
980
981 In addition, when providing custom subclasses of Process objects, you can override the following methods:
982
983 CommandLineTool.make_job_runner
984 ::
985
986 make_job_runner(RuntimeContext)
987 (RuntimeContext) -> Type[JobBase]
988
989 Create and return a job runner object (this implements concrete execution of a command line tool).
990
991 Workflow.make_workflow_step
992 ::
993
994 make_workflow_step(toolpath_object, pos, loadingContext, parentworkflowProv)
995 (Dict[str, Any], int, LoadingContext, Optional[ProvenanceProfile]) -> WorkflowStep
996
997 Create and return a workflow step object.