| 1 |
Metadata-Version: 2.4 |
| 2 |
Name: cwltool |
| 3 |
Version: 3.2.20260413085819 |
| 4 |
Summary: Common workflow language reference implementation |
| 5 |
Home-page: https://github.com/common-workflow-language/cwltool |
| 6 |
Download-URL: https://github.com/common-workflow-language/cwltool |
| 7 |
Author: Common workflow language working group |
| 8 |
Author-email: [email protected] |
| 9 |
Classifier: Development Status :: 5 - Production/Stable |
| 10 |
Classifier: Environment :: Console |
| 11 |
Classifier: Intended Audience :: Developers |
| 12 |
Classifier: Intended Audience :: Science/Research |
| 13 |
Classifier: Intended Audience :: Healthcare Industry |
| 14 |
Classifier: License :: OSI Approved :: Apache Software License |
| 15 |
Classifier: Natural Language :: English |
| 16 |
Classifier: Operating System :: MacOS :: MacOS X |
| 17 |
Classifier: Operating System :: POSIX |
| 18 |
Classifier: Operating System :: POSIX :: Linux |
| 19 |
Classifier: Programming Language :: Python :: 3 |
| 20 |
Classifier: Programming Language :: Python :: 3.10 |
| 21 |
Classifier: Programming Language :: Python :: 3.11 |
| 22 |
Classifier: Programming Language :: Python :: 3.12 |
| 23 |
Classifier: Programming Language :: Python :: 3.13 |
| 24 |
Classifier: Programming Language :: Python :: 3.14 |
| 25 |
Classifier: Topic :: Scientific/Engineering |
| 26 |
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics |
| 27 |
Classifier: Topic :: Scientific/Engineering :: Astronomy |
| 28 |
Classifier: Topic :: Scientific/Engineering :: Atmospheric Science |
| 29 |
Classifier: Topic :: Scientific/Engineering :: Information Analysis |
| 30 |
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps. |
| 31 |
Classifier: Topic :: System :: Distributed Computing |
| 32 |
Classifier: Topic :: Utilities |
| 33 |
Requires-Python: >=3.10, <3.15 |
| 34 |
Description-Content-Type: text/x-rst |
| 35 |
License-File: LICENSE.txt |
| 36 |
Requires-Dist: requests>=2.6.1 |
| 37 |
Requires-Dist: ruamel.yaml<0.20,>=0.16 |
| 38 |
Requires-Dist: rdflib<7.7.0,>=4.2.2 |
| 39 |
Requires-Dist: schema-salad<9,>=8.9 |
| 40 |
Requires-Dist: prov==1.5.1 |
| 41 |
Requires-Dist: mypy-extensions |
| 42 |
Requires-Dist: psutil>=5.6.6 |
| 43 |
Requires-Dist: coloredlogs |
| 44 |
Requires-Dist: pydot>=1.4.1 |
| 45 |
Requires-Dist: argcomplete>=1.12.0 |
| 46 |
Requires-Dist: pyparsing!=3.0.2 |
| 47 |
Requires-Dist: cwl-utils>=0.41 |
| 48 |
Requires-Dist: spython>=0.3.0 |
| 49 |
Requires-Dist: rich-argparse |
| 50 |
Requires-Dist: pygments>=2.20.0 |
| 51 |
Requires-Dist: typing-extensions>=4.1.0 |
| 52 |
Provides-Extra: deps |
| 53 |
Requires-Dist: galaxy-tool-util!=23.0.1,!=23.0.2,!=23.0.3,!=23.0.4,!=23.0.5,<26.1,>=22.1.2; extra == "deps" |
| 54 |
Requires-Dist: galaxy-util<26.1; extra == "deps" |
| 55 |
Requires-Dist: pillow; extra == "deps" |
| 56 |
Dynamic: author |
| 57 |
Dynamic: author-email |
| 58 |
Dynamic: classifier |
| 59 |
Dynamic: description |
| 60 |
Dynamic: description-content-type |
| 61 |
Dynamic: download-url |
| 62 |
Dynamic: home-page |
| 63 |
Dynamic: license-file |
| 64 |
Dynamic: provides-extra |
| 65 |
Dynamic: requires-dist |
| 66 |
Dynamic: requires-python |
| 67 |
Dynamic: summary |
| 68 |
|
| 69 |
############################################################################################# |
| 70 |
``cwltool``: The reference implementation of the Common Workflow Language standards |
| 71 |
############################################################################################# |
| 72 |
|
| 73 |
|Linux Status| |Coverage Status| |Docs Status| |
| 74 |
|
| 75 |
PyPI: |PyPI Version| |PyPI Downloads Month| |Total PyPI Downloads| |
| 76 |
|
| 77 |
Conda: |Conda Version| |Conda Installs| |
| 78 |
|
| 79 |
Debian: |Debian Testing package| |Debian Stable package| |
| 80 |
|
| 81 |
Quay.io (Docker): |Quay.io Container| |
| 82 |
|
| 83 |
.. |Linux Status| image:: https://github.com/common-workflow-language/cwltool/actions/workflows/ci-tests.yml/badge.svg?branch=main |
| 84 |
:target: https://github.com/common-workflow-language/cwltool/actions/workflows/ci-tests.yml |
| 85 |
|
| 86 |
.. |Debian Stable package| image:: https://badges.debian.net/badges/debian/stable/cwltool/version.svg |
| 87 |
:target: https://packages.debian.org/stable/cwltool |
| 88 |
|
| 89 |
.. |Debian Testing package| image:: https://badges.debian.net/badges/debian/testing/cwltool/version.svg |
| 90 |
:target: https://packages.debian.org/testing/cwltool |
| 91 |
|
| 92 |
.. |Coverage Status| image:: https://img.shields.io/codecov/c/github/common-workflow-language/cwltool.svg |
| 93 |
:target: https://codecov.io/gh/common-workflow-language/cwltool |
| 94 |
|
| 95 |
.. |PyPI Version| image:: https://badge.fury.io/py/cwltool.svg |
| 96 |
:target: https://badge.fury.io/py/cwltool |
| 97 |
|
| 98 |
.. |PyPI Downloads Month| image:: https://pepy.tech/badge/cwltool/month |
| 99 |
:target: https://pepy.tech/project/cwltool |
| 100 |
|
| 101 |
.. |Total PyPI Downloads| image:: https://static.pepy.tech/personalized-badge/cwltool?period=total&units=international_system&left_color=black&right_color=orange&left_text=Total%20PyPI%20Downloads |
| 102 |
:target: https://pepy.tech/project/cwltool |
| 103 |
|
| 104 |
.. |Conda Version| image:: https://anaconda.org/conda-forge/cwltool/badges/version.svg |
| 105 |
:target: https://anaconda.org/conda-forge/cwltool |
| 106 |
|
| 107 |
.. |Conda Installs| image:: https://anaconda.org/conda-forge/cwltool/badges/downloads.svg |
| 108 |
:target: https://anaconda.org/conda-forge/cwltool |
| 109 |
|
| 110 |
.. |Quay.io Container| image:: https://quay.io/repository/commonwl/cwltool/status |
| 111 |
:target: https://quay.io/repository/commonwl/cwltool |
| 112 |
|
| 113 |
.. |Docs Status| image:: https://readthedocs.org/projects/cwltool/badge/?version=latest |
| 114 |
:target: https://cwltool.readthedocs.io/en/latest/?badge=latest |
| 115 |
:alt: Documentation Status |
| 116 |
|
| 117 |
This is the reference implementation of the `Common Workflow Language open |
| 118 |
standards <https://www.commonwl.org/>`_. It is intended to be feature complete |
| 119 |
and provide comprehensive validation of CWL |
| 120 |
files as well as provide other tools related to working with CWL. |
| 121 |
|
| 122 |
``cwltool`` is written and tested for |
| 123 |
`Python <https://www.python.org/>`_ ``3.x {x = 9, 10, 11, 12, 13}`` |
| 124 |
|
| 125 |
The reference implementation consists of two packages. The ``cwltool`` package |
| 126 |
is the primary Python module containing the reference implementation in the |
| 127 |
``cwltool`` module and console executable by the same name. |
| 128 |
|
| 129 |
The ``cwlref-runner`` package is optional and provides an additional entry point |
| 130 |
under the alias ``cwl-runner``, which is the implementation-agnostic name for the |
| 131 |
default CWL interpreter installed on a host. |
| 132 |
|
| 133 |
``cwltool`` is provided by the CWL project, `a member project of Software Freedom Conservancy <https://sfconservancy.org/news/2018/apr/11/cwl-new-member-project/>`_ |
| 134 |
and our `many contributors <https://github.com/common-workflow-language/cwltool/graphs/contributors>`_. |
| 135 |
|
| 136 |
.. contents:: Table of Contents |
| 137 |
|
| 138 |
******* |
| 139 |
Install |
| 140 |
******* |
| 141 |
|
| 142 |
``cwltool`` packages |
| 143 |
==================== |
| 144 |
|
| 145 |
Your operating system may offer cwltool directly. For `Debian <https://tracker.debian.org/pkg/cwltool>`_, `Ubuntu <https://launchpad.net/ubuntu/+source/cwltool>`_, |
| 146 |
and similar Linux distribution try |
| 147 |
|
| 148 |
.. code:: bash |
| 149 |
|
| 150 |
sudo apt-get install cwltool |
| 151 |
|
| 152 |
If you encounter an error, first try to update package information by using |
| 153 |
|
| 154 |
.. code:: bash |
| 155 |
|
| 156 |
sudo apt-get update |
| 157 |
|
| 158 |
If you are running macOS X or other UNIXes and you want to use packages prepared by the conda-forge project, then |
| 159 |
please follow the install instructions for `conda-forge <https://conda-forge.org/#about>`_ (if you haven't already) and then |
| 160 |
|
| 161 |
.. code:: bash |
| 162 |
|
| 163 |
conda install -c conda-forge cwltool |
| 164 |
|
| 165 |
All of the above methods of installing ``cwltool`` use packages that might contain bugs already fixed in newer versions or be missing desired features. |
| 166 |
If the packaged version of ``cwltool`` available to you is too old, then we recommend installing using ``pip`` and ``venv`` |
| 167 |
|
| 168 |
.. code:: bash |
| 169 |
|
| 170 |
python3 -m venv env # Create a virtual environment named 'env' in the current directory |
| 171 |
source env/bin/activate # Activate environment before installing `cwltool` |
| 172 |
|
| 173 |
Then install the latest ``cwlref-runner`` package from PyPi (which will install the latest ``cwltool`` package as |
| 174 |
well) |
| 175 |
|
| 176 |
.. code:: bash |
| 177 |
|
| 178 |
pip install cwlref-runner |
| 179 |
|
| 180 |
If installing alongside another CWL implementation (like ``toil-cwl-runner`` or ``arvados-cwl-runner``) then instead run |
| 181 |
|
| 182 |
.. code:: bash |
| 183 |
|
| 184 |
pip install cwltool |
| 185 |
|
| 186 |
MS Windows users |
| 187 |
================ |
| 188 |
|
| 189 |
1. `Install Windows Subsystem for Linux 2 and Docker Desktop <https://docs.docker.com/docker-for-windows/wsl/#prerequisites>`_. |
| 190 |
2. `Install Debian from the Microsoft Store <https://www.microsoft.com/en-us/p/debian/9msvkqc78pk6>`_. |
| 191 |
3. Set Debian as your default WSL 2 distro: ``wsl --set-default debian``. |
| 192 |
4. Return to the Docker Desktop, choose ``Settings`` → ``Resources`` → ``WSL Integration`` and under "Enable integration with additional distros" select "Debian", |
| 193 |
5. Reboot if you have not yet already. |
| 194 |
6. Launch Debian and follow the Linux instructions above (``apt-get install cwltool`` or use the ``venv`` method) |
| 195 |
|
| 196 |
Network problems from within WSL2? Try `these instructions <https://github.com/microsoft/WSL/issues/4731#issuecomment-702176954>`_ followed by ``wsl --shutdown``. |
| 197 |
|
| 198 |
``cwltool`` development version |
| 199 |
=============================== |
| 200 |
|
| 201 |
Or you can skip the direct ``pip`` commands above and install the latest development version of ``cwltool``: |
| 202 |
|
| 203 |
.. code:: bash |
| 204 |
|
| 205 |
git clone https://github.com/common-workflow-language/cwltool.git # clone (copy) the cwltool git repository |
| 206 |
cd cwltool # Change to source directory that git clone just downloaded |
| 207 |
pip install .[deps] # Installs ``cwltool`` from source |
| 208 |
cwltool --version # Check if the installation works correctly |
| 209 |
|
| 210 |
Remember, if co-installing multiple CWL implementations, then you need to |
| 211 |
maintain which implementation ``cwl-runner`` points to via a symbolic file |
| 212 |
system link or `another facility <https://wiki.debian.org/DebianAlternatives>`_. |
| 213 |
|
| 214 |
Recommended Software |
| 215 |
==================== |
| 216 |
|
| 217 |
We strongly suggested to have the following installed: |
| 218 |
|
| 219 |
* One of the following software container engines |
| 220 |
|
| 221 |
* `Podman <https://podman.io/getting-started/installation>`_ |
| 222 |
* `Docker <https://docs.docker.com/engine/install/>`_ |
| 223 |
* Singularity/Apptainer: See `Using Singularity`_ |
| 224 |
* udocker: See `Using uDocker`_ |
| 225 |
|
| 226 |
* `node.js <https://nodejs.org/en/download/>`_ for evaluating CWL Expressions quickly |
| 227 |
(required for `udocker` users, optional but recommended for the other container engines). |
| 228 |
|
| 229 |
Without these, some examples in the CWL tutorials at http://www.commonwl.org/user_guide/ may not work. |
| 230 |
|
| 231 |
*********************** |
| 232 |
Run on the command line |
| 233 |
*********************** |
| 234 |
|
| 235 |
Simple command:: |
| 236 |
|
| 237 |
cwl-runner my_workflow.cwl my_inputs.yaml |
| 238 |
|
| 239 |
Or if you have multiple CWL implementations installed and you want to override |
| 240 |
the default cwl-runner then use:: |
| 241 |
|
| 242 |
cwltool my_workflow.cwl my_inputs.yml |
| 243 |
|
| 244 |
You can set cwltool options in the environment with ``CWLTOOL_OPTIONS``, |
| 245 |
these will be inserted at the beginning of the command line:: |
| 246 |
|
| 247 |
export CWLTOOL_OPTIONS="--debug" |
| 248 |
|
| 249 |
Use with boot2docker on macOS |
| 250 |
============================= |
| 251 |
boot2docker runs Docker inside a virtual machine, and it only mounts ``Users`` |
| 252 |
on it. The default behavior of CWL is to create temporary directories under e.g. |
| 253 |
``/Var`` which is not accessible to Docker containers. |
| 254 |
|
| 255 |
To run CWL successfully with boot2docker you need to set the ``--tmpdir-prefix`` |
| 256 |
and ``--tmp-outdir-prefix`` to somewhere under ``/Users``:: |
| 257 |
|
| 258 |
$ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json |
| 259 |
|
| 260 |
|
| 261 |
Docker default platform on macOS with Apple Silicon |
| 262 |
=================================================== |
| 263 |
|
| 264 |
If macOS users want to run CWL tools/workflows using ``cwltool`` with Docker and their software containers only have support for amd64 (64-bit x86) CPUs, but they have an Apple Silicon (aarch64/arm64) CPU, |
| 265 |
they run into the error: |
| 266 |
|
| 267 |
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested. |
| 268 |
|
| 269 |
To fix this, export the following environment variable before executing `cwltool`: |
| 270 |
|
| 271 |
``export DOCKER_DEFAULT_PLATFORM=linux/amd64`` |
| 272 |
|
| 273 |
To automatically have this variable set in the future, add the same command to ones respective shell profile (e.g. ``~/.zshrc``, ``~/.bash_profile``). |
| 274 |
|
| 275 |
|
| 276 |
Using uDocker |
| 277 |
============= |
| 278 |
|
| 279 |
Some shared computing environments don't support Docker software containers for technical or policy reasons. |
| 280 |
As a workaround, the CWL reference runner supports using the `udocker <https://github.com/indigo-dc/udocker>`_ |
| 281 |
program on Linux using ``--udocker``. |
| 282 |
|
| 283 |
udocker installation: https://indigo-dc.github.io/udocker/installation_manual.html |
| 284 |
|
| 285 |
Run `cwltool` just as you usually would, but with ``--udocker`` prior to the workflow path: |
| 286 |
|
| 287 |
.. code:: bash |
| 288 |
|
| 289 |
cwltool --udocker https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/empty.json |
| 290 |
|
| 291 |
As was mentioned in the `Recommended Software`_ section, |
| 292 |
|
| 293 |
Using Singularity |
| 294 |
================= |
| 295 |
|
| 296 |
``cwltool`` can also use `Singularity <https://github.com/hpcng/singularity/releases/>`_ version 2.6.1 |
| 297 |
or later as a Docker container runtime. |
| 298 |
``cwltool`` with Singularity will run software containers specified in |
| 299 |
``DockerRequirement`` and therefore works with Docker images only, native |
| 300 |
Singularity images are not supported. To use Singularity as the Docker container |
| 301 |
runtime, provide ``--singularity`` command line option to ``cwltool``. |
| 302 |
With Singularity, ``cwltool`` can pass all CWL v1.0 conformance tests, except |
| 303 |
those involving Docker container ENTRYPOINTs. |
| 304 |
|
| 305 |
Example |
| 306 |
|
| 307 |
.. code:: bash |
| 308 |
|
| 309 |
cwltool --singularity https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/cat3-tool-mediumcut.cwl https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/cat-job.json |
| 310 |
|
| 311 |
Running a tool or workflow from remote or local locations |
| 312 |
========================================================= |
| 313 |
|
| 314 |
``cwltool`` can run tool and workflow descriptions on both local and remote |
| 315 |
systems via its support for HTTP[S] URLs. |
| 316 |
|
| 317 |
Input job files and Workflow steps (via the `run` directive) can reference CWL |
| 318 |
documents using absolute or relative local filesystem paths. If a relative path |
| 319 |
is referenced and that document isn't found in the current directory, then the |
| 320 |
following locations will be searched: |
| 321 |
http://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem |
| 322 |
|
| 323 |
You can also use `cwldep <https://github.com/common-workflow-language/cwldep>`_ |
| 324 |
to manage dependencies on external tools and workflows. |
| 325 |
|
| 326 |
Overriding workflow requirements at load time |
| 327 |
============================================= |
| 328 |
|
| 329 |
Sometimes a workflow needs additional requirements to run in a particular |
| 330 |
environment or with a particular dataset. To avoid the need to modify the |
| 331 |
underlying workflow, cwltool supports requirement "overrides". |
| 332 |
|
| 333 |
The format of the "overrides" object is a mapping of item identifier (workflow, |
| 334 |
workflow step, or command line tool) to the process requirements that should be applied. |
| 335 |
|
| 336 |
.. code:: yaml |
| 337 |
|
| 338 |
cwltool:overrides: |
| 339 |
echo.cwl: |
| 340 |
requirements: |
| 341 |
EnvVarRequirement: |
| 342 |
envDef: |
| 343 |
MESSAGE: override_value |
| 344 |
|
| 345 |
Overrides can be specified either on the command line, or as part of the job |
| 346 |
input document. Workflow steps are identified using the name of the workflow |
| 347 |
file followed by the step name as a document fragment identifier "#id". |
| 348 |
Override identifiers are relative to the top-level workflow document. |
| 349 |
|
| 350 |
.. code:: bash |
| 351 |
|
| 352 |
cwltool --overrides overrides.yml my-tool.cwl my-job.yml |
| 353 |
|
| 354 |
.. code:: yaml |
| 355 |
|
| 356 |
input_parameter1: value1 |
| 357 |
input_parameter2: value2 |
| 358 |
cwltool:overrides: |
| 359 |
workflow.cwl#step1: |
| 360 |
requirements: |
| 361 |
EnvVarRequirement: |
| 362 |
envDef: |
| 363 |
MESSAGE: override_value |
| 364 |
|
| 365 |
.. code:: bash |
| 366 |
|
| 367 |
cwltool my-tool.cwl my-job-with-overrides.yml |
| 368 |
|
| 369 |
|
| 370 |
Combining parts of a workflow into a single document |
| 371 |
==================================================== |
| 372 |
|
| 373 |
Use ``--pack`` to combine a workflow made up of multiple files into a |
| 374 |
single compound document. This operation takes all the CWL files |
| 375 |
referenced by a workflow and builds a new CWL document with all |
| 376 |
Process objects (CommandLineTool and Workflow) in a list in the |
| 377 |
``$graph`` field. Cross references (such as ``run:`` and ``source:`` |
| 378 |
fields) are updated to internal references within the new packed |
| 379 |
document. The top-level workflow is named ``#main``. |
| 380 |
|
| 381 |
.. code:: bash |
| 382 |
|
| 383 |
cwltool --pack my-wf.cwl > my-packed-wf.cwl |
| 384 |
|
| 385 |
|
| 386 |
Running only part of a workflow |
| 387 |
=============================== |
| 388 |
|
| 389 |
You can run a partial workflow with the ``--target`` (``-t``) option. This |
| 390 |
takes the name of an output parameter, workflow step, or input |
| 391 |
parameter in the top-level workflow. You may provide multiple |
| 392 |
targets. |
| 393 |
|
| 394 |
.. code:: bash |
| 395 |
|
| 396 |
cwltool --target step3 my-wf.cwl |
| 397 |
|
| 398 |
If a target is an output parameter, it will only run only the steps |
| 399 |
that contribute to that output. If a target is a workflow step, it |
| 400 |
will run the workflow starting from that step. If a target is an |
| 401 |
input parameter, it will only run the steps connected to |
| 402 |
that input. |
| 403 |
|
| 404 |
Use ``--print-targets`` to get a listing of the targets of a workflow. |
| 405 |
To see which steps will run, use ``--print-subgraph`` with |
| 406 |
``--target`` to get a printout of the workflow subgraph for the |
| 407 |
selected targets. |
| 408 |
|
| 409 |
.. code:: bash |
| 410 |
|
| 411 |
cwltool --print-targets my-wf.cwl |
| 412 |
|
| 413 |
cwltool --target step3 --print-subgraph my-wf.cwl > my-wf-starting-from-step3.cwl |
| 414 |
|
| 415 |
|
| 416 |
Visualizing a CWL document |
| 417 |
========================== |
| 418 |
|
| 419 |
The ``--print-dot`` option will print a file suitable for Graphviz ``dot`` program. Here is a bash onliner to generate a Scalable Vector Graphic (SVG) file: |
| 420 |
|
| 421 |
.. code:: bash |
| 422 |
|
| 423 |
cwltool --print-dot my-wf.cwl | dot -Tsvg > my-wf.svg |
| 424 |
|
| 425 |
Modeling a CWL document as RDF |
| 426 |
============================== |
| 427 |
|
| 428 |
CWL documents can be expressed as RDF triple graphs. |
| 429 |
|
| 430 |
.. code:: bash |
| 431 |
|
| 432 |
cwltool --print-rdf --rdf-serializer=turtle mywf.cwl |
| 433 |
|
| 434 |
|
| 435 |
Environment Variables in cwltool |
| 436 |
================================ |
| 437 |
|
| 438 |
This reference implementation supports several ways of setting |
| 439 |
environment variables for tools, in addition to the standard |
| 440 |
``EnvVarRequirement``. The sequence of steps applied to create the |
| 441 |
environment is: |
| 442 |
|
| 443 |
0. If the ``--preserve-entire-environment`` flag is present, then begin with the current |
| 444 |
environment, else begin with an empty environment. |
| 445 |
|
| 446 |
1. Add any variables specified by ``--preserve-environment`` option(s). |
| 447 |
|
| 448 |
2. Set ``TMPDIR`` and ``HOME`` per `the CWL v1.0+ CommandLineTool specification <https://www.commonwl.org/v1.0/CommandLineTool.html#Runtime_environment>`_. |
| 449 |
|
| 450 |
3. Apply any ``EnvVarRequirement`` from the ``CommandLineTool`` description. |
| 451 |
|
| 452 |
4. Apply any manipulations required by any ``cwltool:MPIRequirement`` extensions. |
| 453 |
|
| 454 |
5. Substitute any secrets required by ``Secrets`` extension. |
| 455 |
|
| 456 |
6. Modify the environment in response to ``SoftwareRequirement`` (see below). |
| 457 |
|
| 458 |
|
| 459 |
Leveraging SoftwareRequirements (Beta) |
| 460 |
-------------------------------------- |
| 461 |
|
| 462 |
CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool |
| 463 |
may in turn use to resolve to packages in various package managers or |
| 464 |
dependency management systems such as `Environment Modules |
| 465 |
<http://modules.sourceforge.net/>`__. |
| 466 |
|
| 467 |
Utilizing ``SoftwareRequirement`` hints using cwltool requires an optional |
| 468 |
dependency, for this reason be sure to use specify the ``deps`` modifier when |
| 469 |
installing cwltool. For instance:: |
| 470 |
|
| 471 |
$ pip install 'cwltool[deps]' |
| 472 |
|
| 473 |
Installing cwltool in this fashion enables several new command line options. |
| 474 |
The most general of these options is ``--beta-dependency-resolvers-configuration``. |
| 475 |
This option allows one to specify a dependency resolver's configuration file. |
| 476 |
This file may be specified as either XML or YAML and very simply describes various |
| 477 |
plugins to enable to "resolve" ``SoftwareRequirement`` dependencies. |
| 478 |
|
| 479 |
Using these hints will allow cwltool to modify the environment in |
| 480 |
which your tool runs, for example by loading one or more Environment |
| 481 |
Modules. The environment is constructed as above, then the environment |
| 482 |
may modified by the selected tool resolver. This currently means that |
| 483 |
you cannot override any environment variables set by the selected tool |
| 484 |
resolver. Note that the environment given to the configured dependency |
| 485 |
resolver has the variable `_CWLTOOL` set to `1` to allow introspection. |
| 486 |
|
| 487 |
To discuss some of these plugins and how to configure them, first consider the |
| 488 |
following ``hint`` definition for an example CWL tool. |
| 489 |
|
| 490 |
.. code:: yaml |
| 491 |
|
| 492 |
SoftwareRequirement: |
| 493 |
packages: |
| 494 |
- package: seqtk |
| 495 |
version: |
| 496 |
- r93 |
| 497 |
|
| 498 |
Now imagine deploying cwltool on a cluster with Software Modules installed |
| 499 |
and that a ``seqtk`` module is available at version ``r93``. This means cluster |
| 500 |
users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after |
| 501 |
sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is |
| 502 |
available on the ``PATH``. A simple dependency resolvers configuration file, called |
| 503 |
``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source |
| 504 |
the correct module environment before executing the above tool would simply be: |
| 505 |
|
| 506 |
.. code:: yaml |
| 507 |
|
| 508 |
- type: modules |
| 509 |
|
| 510 |
The outer list indicates that one plugin is being enabled, the plugin parameters are |
| 511 |
defined as a dictionary for this one list item. There is only one required parameter |
| 512 |
for the plugin above, this is ``type`` and defines the plugin type. This parameter |
| 513 |
is required for all plugins. The available plugins and the parameters |
| 514 |
available for each are documented (incompletely) `here |
| 515 |
<https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__. |
| 516 |
Unfortunately, this documentation is in the context of Galaxy tool |
| 517 |
``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly. |
| 518 |
|
| 519 |
cwltool is distributed with an example of such seqtk tool and sample corresponding |
| 520 |
job. It could executed from the cwltool root using a dependency resolvers |
| 521 |
configuration file such as the above one using the command:: |
| 522 |
|
| 523 |
cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \ |
| 524 |
tests/seqtk_seq.cwl \ |
| 525 |
tests/seqtk_seq_job.json |
| 526 |
|
| 527 |
This example demonstrates both that cwltool can leverage |
| 528 |
existing software installations and also handle workflows with dependencies |
| 529 |
on different versions of the same software and libraries. However the above |
| 530 |
example does require an existing module setup so it is impossible to test this example |
| 531 |
"out of the box" with cwltool. For a more isolated test that demonstrates all |
| 532 |
the same concepts - the resolver plugin type ``galaxy_packages`` can be used. |
| 533 |
|
| 534 |
"Galaxy packages" are a lighter-weight alternative to Environment Modules that are |
| 535 |
really just defined by a way to lay out directories into packages and versions |
| 536 |
to find little scripts that are sourced to modify the environment. They have |
| 537 |
been used for years in Galaxy community to adapt Galaxy tools to cluster |
| 538 |
environments but require neither knowledge of Galaxy nor any special tools to |
| 539 |
setup. These should work just fine for CWL tools. |
| 540 |
|
| 541 |
The cwltool source code repository's test directory is setup with a very simple |
| 542 |
directory that defines a set of "Galaxy packages" (but really just defines one |
| 543 |
package named ``random-lines``). The directory layout is simply:: |
| 544 |
|
| 545 |
tests/test_deps_env/ |
| 546 |
random-lines/ |
| 547 |
1.0/ |
| 548 |
env.sh |
| 549 |
|
| 550 |
If the ``galaxy_packages`` plugin is enabled and pointed at the |
| 551 |
``tests/test_deps_env`` directory in cwltool's root and a ``SoftwareRequirement`` |
| 552 |
such as the following is encountered. |
| 553 |
|
| 554 |
.. code:: yaml |
| 555 |
|
| 556 |
hints: |
| 557 |
SoftwareRequirement: |
| 558 |
packages: |
| 559 |
- package: 'random-lines' |
| 560 |
version: |
| 561 |
- '1.0' |
| 562 |
|
| 563 |
Then cwltool will simply find that ``env.sh`` file and source it before executing |
| 564 |
the corresponding tool. That ``env.sh`` script is only responsible for modifying |
| 565 |
the job's ``PATH`` to add the required binaries. |
| 566 |
|
| 567 |
This is a full example that works since resolving "Galaxy packages" has no |
| 568 |
external requirements. Try it out by executing the following command from cwltool's |
| 569 |
root directory:: |
| 570 |
|
| 571 |
cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf.yml \ |
| 572 |
tests/random_lines.cwl \ |
| 573 |
tests/random_lines_job.json |
| 574 |
|
| 575 |
The resolvers configuration file in the above example was simply: |
| 576 |
|
| 577 |
.. code:: yaml |
| 578 |
|
| 579 |
- type: galaxy_packages |
| 580 |
base_path: ./tests/test_deps_env |
| 581 |
|
| 582 |
It is possible that the ``SoftwareRequirement`` s in a given CWL tool will not |
| 583 |
match the module names for a given cluster. Such requirements can be re-mapped |
| 584 |
to specific deployed packages or versions using another file specified using |
| 585 |
the resolver plugin parameter `mapping_files`. We will |
| 586 |
demonstrate this using `galaxy_packages,` but the concepts apply equally well |
| 587 |
to Environment Modules or Conda packages (described below), for instance. |
| 588 |
|
| 589 |
So consider the resolvers configuration file. |
| 590 |
(`tests/test_deps_env_resolvers_conf_rewrite.yml`): |
| 591 |
|
| 592 |
.. code:: yaml |
| 593 |
|
| 594 |
- type: galaxy_packages |
| 595 |
base_path: ./tests/test_deps_env |
| 596 |
mapping_files: ./tests/test_deps_mapping.yml |
| 597 |
|
| 598 |
And the corresponding mapping configuration file (`tests/test_deps_mapping.yml`): |
| 599 |
|
| 600 |
.. code:: yaml |
| 601 |
|
| 602 |
- from: |
| 603 |
name: randomLines |
| 604 |
version: 1.0.0-rc1 |
| 605 |
to: |
| 606 |
name: random-lines |
| 607 |
version: '1.0' |
| 608 |
|
| 609 |
This is saying if cwltool encounters a requirement of ``randomLines`` at version |
| 610 |
``1.0.0-rc1`` in a tool, to rewrite to our specific plugin as ``random-lines`` at |
| 611 |
version ``1.0``. cwltool has such a test tool called ``random_lines_mapping.cwl`` |
| 612 |
that contains such a source ``SoftwareRequirement``. To try out this example with |
| 613 |
mapping, execute the following command from the cwltool root directory:: |
| 614 |
|
| 615 |
cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf_rewrite.yml \ |
| 616 |
tests/random_lines_mapping.cwl \ |
| 617 |
tests/random_lines_job.json |
| 618 |
|
| 619 |
The previous examples demonstrated leveraging existing infrastructure to |
| 620 |
provide requirements for CWL tools. If instead a real package manager is used |
| 621 |
cwltool has the opportunity to install requirements as needed. While initial |
| 622 |
support for Homebrew/Linuxbrew plugins is available, the most developed such |
| 623 |
plugin is for the `Conda <https://conda.io/docs/#>`__ package manager. Conda has the nice properties |
| 624 |
of allowing multiple versions of a package to be installed simultaneously, |
| 625 |
not requiring evaluated permissions to install Conda itself or packages using |
| 626 |
Conda, and being cross-platform. For these reasons, cwltool may run as a normal |
| 627 |
user, install its own Conda environment and manage multiple versions of Conda packages |
| 628 |
on Linux and Mac OS X. |
| 629 |
|
| 630 |
The Conda plugin can be endlessly configured, but a sensible set of defaults |
| 631 |
that has proven a powerful stack for dependency management within the Galaxy tool |
| 632 |
development ecosystem can be enabled by simply passing cwltool the |
| 633 |
``--beta-conda-dependencies`` flag. |
| 634 |
|
| 635 |
With this, we can use the seqtk example above without Docker or any externally managed services - cwltool should install everything it needs |
| 636 |
and create an environment for the tool. Try it out with the following command:: |
| 637 |
|
| 638 |
cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json |
| 639 |
|
| 640 |
The CWL specification allows URIs to be attached to ``SoftwareRequirement`` s |
| 641 |
that allow disambiguation of package names. If the mapping files described above |
| 642 |
allow deployers to adapt tools to their infrastructure, this mechanism allows |
| 643 |
tools to adapt their requirements to multiple package managers. To demonstrate |
| 644 |
this within the context of the seqtk, we can simply break the package name we |
| 645 |
use and then specify a specific Conda package as follows: |
| 646 |
|
| 647 |
.. code:: yaml |
| 648 |
|
| 649 |
hints: |
| 650 |
SoftwareRequirement: |
| 651 |
packages: |
| 652 |
- package: seqtk_seq |
| 653 |
version: |
| 654 |
- '1.2' |
| 655 |
specs: |
| 656 |
- https://anaconda.org/bioconda/seqtk |
| 657 |
- https://packages.debian.org/sid/seqtk |
| 658 |
|
| 659 |
The example can be executed using the command:: |
| 660 |
|
| 661 |
cwltool --beta-conda-dependencies tests/seqtk_seq_wrong_name.cwl tests/seqtk_seq_job.json |
| 662 |
|
| 663 |
The plugin framework for managing the resolution of these software requirements |
| 664 |
as maintained as part of `galaxy-tool-util <https://github.com/galaxyproject/galaxy/tree/dev/packages/tool_util>`__ - a small, |
| 665 |
portable subset of the Galaxy project. More information on configuration and implementation can be found |
| 666 |
at the following links: |
| 667 |
|
| 668 |
- `Dependency Resolvers in Galaxy <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__ |
| 669 |
- `Conda for [Galaxy] Tool Dependencies <https://docs.galaxyproject.org/en/latest/admin/conda_faq.html>`__ |
| 670 |
- `Mapping Files - Implementation <https://github.com/galaxyproject/galaxy/commit/495802d229967771df5b64a2f79b88a0eaf00edb>`__ |
| 671 |
- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__ |
| 672 |
- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__ |
| 673 |
|
| 674 |
Use with GA4GH Tool Registry API |
| 675 |
================================ |
| 676 |
|
| 677 |
Cwltool can launch tools directly from `GA4GH Tool Registry API`_ endpoints. |
| 678 |
|
| 679 |
By default, cwltool searches https://dockstore.org/ . Use ``--add-tool-registry`` to add other registries to the search path. |
| 680 |
|
| 681 |
For example :: |
| 682 |
|
| 683 |
cwltool quay.io/collaboratory/dockstore-tool-bamstats:develop test.json |
| 684 |
|
| 685 |
and (defaults to latest when a version is not specified) :: |
| 686 |
|
| 687 |
cwltool quay.io/collaboratory/dockstore-tool-bamstats test.json |
| 688 |
|
| 689 |
For this example, grab the test.json (and input file) from https://github.com/CancerCollaboratory/dockstore-tool-bamstats :: |
| 690 |
|
| 691 |
wget https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-bamstats/versions/develop/PLAIN-CWL/descriptor/test.json |
| 692 |
wget https://github.com/CancerCollaboratory/dockstore-tool-bamstats/raw/develop/rna.SRR948778.bam |
| 693 |
|
| 694 |
|
| 695 |
.. _`GA4GH Tool Registry API`: https://github.com/ga4gh/tool-registry-schemas |
| 696 |
|
| 697 |
Running MPI-based tools that need to be launched |
| 698 |
================================================ |
| 699 |
|
| 700 |
Cwltool supports an extension to the CWL spec |
| 701 |
``http://commonwl.org/cwltool#MPIRequirement``. When the tool |
| 702 |
definition has this in its ``requirements``/``hints`` section, and |
| 703 |
cwltool has been run with ``--enable-ext``, then the tool's command |
| 704 |
line will be extended with the commands needed to launch it with |
| 705 |
``mpirun`` or similar. You can specify the number of processes to |
| 706 |
start as either a literal integer or an expression (that will result |
| 707 |
in an integer). For example:: |
| 708 |
|
| 709 |
#!/usr/bin/env cwl-runner |
| 710 |
cwlVersion: v1.1 |
| 711 |
class: CommandLineTool |
| 712 |
$namespaces: |
| 713 |
cwltool: "http://commonwl.org/cwltool#" |
| 714 |
requirements: |
| 715 |
cwltool:MPIRequirement: |
| 716 |
processes: $(inputs.nproc) |
| 717 |
inputs: |
| 718 |
nproc: |
| 719 |
type: int |
| 720 |
|
| 721 |
Interaction with containers: the MPIRequirement currently prepends its |
| 722 |
commands to the front of the command line that is constructed. If you |
| 723 |
wish to run a containerized application in parallel, for simple use |
| 724 |
cases, this does work with Singularity, depending upon the platform |
| 725 |
setup. However, this combination should be considered "alpha" -- please |
| 726 |
do report any issues you have! This does not work with Docker at the |
| 727 |
moment. (More precisely, you get `n` copies of the same single process |
| 728 |
image run at the same time that cannot communicate with each other.) |
| 729 |
|
| 730 |
The host-specific parameters are configured in a simple YAML file |
| 731 |
(specified with the ``--mpi-config-file`` flag). The allowed keys are |
| 732 |
given in the following table; all are optional. |
| 733 |
|
| 734 |
+----------------+------------------+----------+------------------------------+ |
| 735 |
| Key | Type | Default | Description | |
| 736 |
+================+==================+==========+==============================+ |
| 737 |
| runner | str | "mpirun" | The primary command to use. | |
| 738 |
+----------------+------------------+----------+------------------------------+ |
| 739 |
| nproc_flag | str | "-n" | Flag to set number of | |
| 740 |
| | | | processes to start. | |
| 741 |
+----------------+------------------+----------+------------------------------+ |
| 742 |
| default_nproc | int | 1 | Default number of processes. | |
| 743 |
+----------------+------------------+----------+------------------------------+ |
| 744 |
| extra_flags | List[str] | [] | A list of any other flags to | |
| 745 |
| | | | be added to the runner's | |
| 746 |
| | | | command line before | |
| 747 |
| | | | the ``baseCommand``. | |
| 748 |
+----------------+------------------+----------+------------------------------+ |
| 749 |
| env_pass | List[str] | [] | A list of environment | |
| 750 |
| | | | variables that should be | |
| 751 |
| | | | passed from the host | |
| 752 |
| | | | environment through to the | |
| 753 |
| | | | tool (e.g., giving the | |
| 754 |
| | | | node list as set by your | |
| 755 |
| | | | scheduler). | |
| 756 |
+----------------+------------------+----------+------------------------------+ |
| 757 |
| env_pass_regex | List[str] | [] | A list of python regular | |
| 758 |
| | | | expressions that will be | |
| 759 |
| | | | matched against the host's | |
| 760 |
| | | | environment. Those that match| |
| 761 |
| | | | will be passed through. | |
| 762 |
+----------------+------------------+----------+------------------------------+ |
| 763 |
| env_set | Mapping[str,str] | {} | A dictionary whose keys are | |
| 764 |
| | | | the environment variables set| |
| 765 |
| | | | and the values being the | |
| 766 |
| | | | values. | |
| 767 |
+----------------+------------------+----------+------------------------------+ |
| 768 |
|
| 769 |
|
| 770 |
Enabling Fast Parser (experimental) |
| 771 |
=================================== |
| 772 |
|
| 773 |
For very large workflows, `cwltool` can spend a lot of time in |
| 774 |
initialization, before the first step runs. There is an experimental |
| 775 |
flag ``--fast-parser`` which can dramatically reduce the |
| 776 |
initialization overhead, however as of this writing it has several limitations: |
| 777 |
|
| 778 |
- Error reporting in general is worse than the standard parser, you will want to use it with workflows that you know are already correct. |
| 779 |
|
| 780 |
- It does not check for dangling links (these will become runtime errors instead of loading errors) |
| 781 |
|
| 782 |
- Several other cases fail, as documented in https://github.com/common-workflow-language/cwltool/pull/1720 |
| 783 |
|
| 784 |
*********** |
| 785 |
Development |
| 786 |
*********** |
| 787 |
|
| 788 |
Running tests locally |
| 789 |
===================== |
| 790 |
|
| 791 |
- Running basic tests ``(/tests)``: |
| 792 |
|
| 793 |
To run the basic tests after installing `cwltool` execute the following: |
| 794 |
|
| 795 |
.. code:: bash |
| 796 |
|
| 797 |
pip install -rtest-requirements.txt |
| 798 |
pytest ## N.B. This requires node.js or docker to be available |
| 799 |
|
| 800 |
To run various tests in all supported Python environments, we use `tox <https://github.com/common-workflow-language/cwltool/tree/main/tox.ini>`_. To run the test suite in all supported Python environments |
| 801 |
first clone the complete code repository (see the ``git clone`` instructions above) and then run |
| 802 |
the following in the terminal: |
| 803 |
``pip install tox; tox -p`` |
| 804 |
|
| 805 |
List of all environment can be seen using: |
| 806 |
``tox --listenvs`` |
| 807 |
and running a specific test env using: |
| 808 |
``tox run -e <env name>`` |
| 809 |
and additionally run a specific test using this format: |
| 810 |
``tox run -e py310-unit -- -v tests/test_examples.py::test_scandeps`` |
| 811 |
|
| 812 |
- Running the entire suite of CWL conformance tests: |
| 813 |
|
| 814 |
The GitHub repository for the CWL specifications contains a script that tests a CWL |
| 815 |
implementation against a wide array of valid CWL files using the `cwltest <https://github.com/common-workflow-language/cwltest>`_ |
| 816 |
program |
| 817 |
|
| 818 |
Instructions for running these tests can be found in the Common Workflow Language Specification repository at https://github.com/common-workflow-language/common-workflow-language/blob/main/CONFORMANCE_TESTS.md . |
| 819 |
|
| 820 |
Import as a module |
| 821 |
================== |
| 822 |
|
| 823 |
Add |
| 824 |
|
| 825 |
.. code:: python |
| 826 |
|
| 827 |
import cwltool |
| 828 |
|
| 829 |
to your script. |
| 830 |
|
| 831 |
The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory |
| 832 |
|
| 833 |
.. code:: python |
| 834 |
|
| 835 |
import cwltool.factory |
| 836 |
fac = cwltool.factory.Factory() |
| 837 |
|
| 838 |
echo = fac.make("echo.cwl") |
| 839 |
result = echo(inp="foo") |
| 840 |
|
| 841 |
# result["out"] == "foo" |
| 842 |
|
| 843 |
|
| 844 |
CWL Tool Control Flow |
| 845 |
===================== |
| 846 |
|
| 847 |
Technical outline of how cwltool works internally, for maintainers. |
| 848 |
|
| 849 |
#. Use CWL ``load_tool()`` to load document. |
| 850 |
|
| 851 |
#. Fetches the document from file or URL |
| 852 |
#. Applies preprocessing (syntax/identifier expansion and normalization) |
| 853 |
#. Validates the document based on cwlVersion |
| 854 |
#. If necessary, updates the document to the latest spec |
| 855 |
#. Constructs a Process object using ``make_tool()``` callback. This yields a |
| 856 |
CommandLineTool, Workflow, or ExpressionTool. For workflows, this |
| 857 |
recursively constructs each workflow step. |
| 858 |
#. To construct custom types for CommandLineTool, Workflow, or |
| 859 |
ExpressionTool, provide a custom ``make_tool()`` |
| 860 |
|
| 861 |
#. Iterate on the ``job()`` method of the Process object to get back runnable jobs. |
| 862 |
|
| 863 |
#. ``job()`` is a generator method (uses the Python iterator protocol) |
| 864 |
#. Each time the ``job()`` method is invoked in an iteration, it returns one |
| 865 |
of: a runnable item (an object with a ``run()`` method), ``None`` (indicating |
| 866 |
there is currently no work ready to run) or end of iteration (indicating |
| 867 |
the process is complete.) |
| 868 |
#. Invoke the runnable item by calling ``run()``. This runs the tool and gets output. |
| 869 |
#. An output callback reports the output of a process. |
| 870 |
#. ``job()`` may be iterated over multiple times. It will yield all the work |
| 871 |
that is currently ready to run and then yield None. |
| 872 |
|
| 873 |
#. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation. |
| 874 |
|
| 875 |
#. The WorkflowJob iterates over each WorkflowJobStep and determines if the |
| 876 |
inputs the step are ready. |
| 877 |
#. When a step is ready, it constructs an input object for that step and |
| 878 |
iterates on the ``job()`` method of the workflow job step. |
| 879 |
#. Each runnable item is yielded back up to top-level run loop |
| 880 |
#. When a step job completes and receives an output callback, the |
| 881 |
job outputs are assigned to the output of the workflow step. |
| 882 |
#. When all steps are complete, the intermediate files are moved to a final |
| 883 |
workflow output, intermediate directories are deleted, and the workflow's output callback is called. |
| 884 |
|
| 885 |
#. ``CommandLineTool`` job() objects yield a single runnable object. |
| 886 |
|
| 887 |
#. The CommandLineTool ``job()`` method calls ``make_job_runner()`` to create a |
| 888 |
``CommandLineJob`` object |
| 889 |
#. The job method configures the CommandLineJob object by setting public |
| 890 |
attributes |
| 891 |
#. The job method iterates over file and directories inputs to the |
| 892 |
CommandLineTool and creates a "path map". |
| 893 |
#. Files are mapped from their "resolved" location to a "target" path where |
| 894 |
they will appear at tool invocation (for example, a location inside a |
| 895 |
Docker container.) The target paths are used on the command line. |
| 896 |
#. Files are staged to targets paths using either Docker volume binds (when |
| 897 |
using containers) or symlinks (if not). This staging step enables files |
| 898 |
to be logically rearranged or renamed independent of their source layout. |
| 899 |
#. The ``run()`` method of CommandLineJob executes the command line tool or |
| 900 |
Docker container, waits for it to complete, collects output, and makes |
| 901 |
the output callback. |
| 902 |
|
| 903 |
Extension points |
| 904 |
================ |
| 905 |
|
| 906 |
The following functions can be passed to main() to override or augment |
| 907 |
the listed behaviors. |
| 908 |
|
| 909 |
executor |
| 910 |
:: |
| 911 |
|
| 912 |
executor(tool, job_order_object, runtimeContext, logger) |
| 913 |
(Process, Dict[str, Any], RuntimeContext) -> Tuple[Dict[str, Any], str] |
| 914 |
|
| 915 |
An implementation of the top-level workflow execution loop should |
| 916 |
synchronously run a process object to completion and return the |
| 917 |
output object. |
| 918 |
|
| 919 |
versionfunc |
| 920 |
:: |
| 921 |
|
| 922 |
() |
| 923 |
() -> str |
| 924 |
|
| 925 |
Return version string. |
| 926 |
|
| 927 |
logger_handler |
| 928 |
:: |
| 929 |
|
| 930 |
logger_handler |
| 931 |
logging.Handler |
| 932 |
|
| 933 |
Handler object for logging. |
| 934 |
|
| 935 |
The following functions can be set in LoadingContext to override or |
| 936 |
augment the listed behaviors. |
| 937 |
|
| 938 |
fetcher_constructor |
| 939 |
:: |
| 940 |
|
| 941 |
fetcher_constructor(cache, session) |
| 942 |
(Dict[unicode, unicode], requests.sessions.Session) -> Fetcher |
| 943 |
|
| 944 |
Construct a Fetcher object with the supplied cache and HTTP session. |
| 945 |
|
| 946 |
resolver |
| 947 |
:: |
| 948 |
|
| 949 |
resolver(document_loader, document) |
| 950 |
(Loader, str | dict[str, Any]) -> str |
| 951 |
|
| 952 |
Resolve a relative document identifier to an absolute one that can be fetched. |
| 953 |
|
| 954 |
The following functions can be set in RuntimeContext to override or |
| 955 |
augment the listed behaviors. |
| 956 |
|
| 957 |
construct_tool_object |
| 958 |
:: |
| 959 |
|
| 960 |
construct_tool_object(toolpath_object, loadingContext) |
| 961 |
(MutableMapping[str, Any], LoadingContext) -> Process |
| 962 |
|
| 963 |
Hook to construct a Process object (eg CommandLineTool) object from a document. |
| 964 |
|
| 965 |
select_resources |
| 966 |
:: |
| 967 |
|
| 968 |
selectResources(request) |
| 969 |
(Dict[str, int], RuntimeContext) -> Dict[str, int] |
| 970 |
|
| 971 |
Take a resource request and turn it into a concrete resource assignment. |
| 972 |
|
| 973 |
make_fs_access |
| 974 |
:: |
| 975 |
|
| 976 |
make_fs_access(basedir) |
| 977 |
(str) -> StdFsAccess |
| 978 |
|
| 979 |
Return a file system access object. |
| 980 |
|
| 981 |
In addition, when providing custom subclasses of Process objects, you can override the following methods: |
| 982 |
|
| 983 |
CommandLineTool.make_job_runner |
| 984 |
:: |
| 985 |
|
| 986 |
make_job_runner(RuntimeContext) |
| 987 |
(RuntimeContext) -> Type[JobBase] |
| 988 |
|
| 989 |
Create and return a job runner object (this implements concrete execution of a command line tool). |
| 990 |
|
| 991 |
Workflow.make_workflow_step |
| 992 |
:: |
| 993 |
|
| 994 |
make_workflow_step(toolpath_object, pos, loadingContext, parentworkflowProv) |
| 995 |
(Dict[str, Any], int, LoadingContext, Optional[ProvenanceProfile]) -> WorkflowStep |
| 996 |
|
| 997 |
Create and return a workflow step object. |