| 1 |
.. -*- mode: rst -*- |
| 2 |
.. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79: |
| 3 |
|
| 4 |
.. _chap_design_testing: |
| 5 |
|
| 6 |
********************************** |
| 7 |
Continuous integration and testing |
| 8 |
********************************** |
| 9 |
|
| 10 |
.. topic:: Specification scope and status |
| 11 |
|
| 12 |
This specification describes the current implementation. |
| 13 |
|
| 14 |
DataLad is tested using a pytest-based testsuite that is run locally and via continuous integrations setups. |
| 15 |
Code development should ensure that old and new functionality is appropriately tested. |
| 16 |
The project aims for good unittest coverage (at least 80%). |
| 17 |
|
| 18 |
Running tests |
| 19 |
============= |
| 20 |
|
| 21 |
|
| 22 |
Starting at the top level with ``datalad/tests``, every module in the package comes with a subdirectory ``tests/``, containing the tests for that portion of the codebase. This structure is meant to simplify (re-)running the tests for a particular module. |
| 23 |
The test suite is run using |
| 24 |
|
| 25 |
.. code-block:: bash |
| 26 |
|
| 27 |
pip install -e .[tests] |
| 28 |
python -m pytest -c tox.ini datalad |
| 29 |
# or, with coverage reports |
| 30 |
python -m pytest -c tox.ini --cov=datalad datalad |
| 31 |
|
| 32 |
Individual tests can be run using a path to the test file, followed by two colons and the test name: |
| 33 |
|
| 34 |
.. code-block:: bash |
| 35 |
|
| 36 |
python -m pytest datalad/core/local/tests/test_save.py::test_save_message_file |
| 37 |
|
| 38 |
The set of to-be-run tests can be further sub-selected with environment variable based configurations that enable tests based on their :ref:`decorators`, or pytest-specific parameters. |
| 39 |
Invoking a test run using ``DATALAD_TESTS_KNOWNFAILURES_PROBE=True pytest datalad``, for example, will run tests marked as known failures whether or not they still fail. |
| 40 |
See section :ref:`configuration` for all available configurations. |
| 41 |
Invoking a test run using ``DATALAD_TESTS_SSH=1 pytest -m xfail -c tox.ini datalad`` will run only those tests marked as `xfail <https://docs.pytest.org/en/latest/how-to/skipping.html>`_. |
| 42 |
|
| 43 |
Local setup |
| 44 |
----------- |
| 45 |
Local test execution usually requires a local installation with all development requirements. It is recommended to either use a `virtualenv <https://virtualenv.pypa.io/en/latest/>`_, or `tox <https://tox.wiki/en/latest/>`_ via a ``tox.ini`` file in the code base. |
| 46 |
|
| 47 |
CI setup |
| 48 |
-------- |
| 49 |
At the moment, Travis-CI, Appveyor, and GitHub Workflows exercise the tests battery for every PR and on the default branch, covering different operating systems, Python versions, and file systems. |
| 50 |
Tests should be ran on the oldest, latest, and current stable Python release. |
| 51 |
The projects uses https://codecov.io for an overview of code coverage. |
| 52 |
|
| 53 |
|
| 54 |
Writing tests |
| 55 |
============= |
| 56 |
|
| 57 |
Additional functionality is tested by extending existing similar tests with new test cases, or adding new tests to the respective test script of the module. Generally, every file `example.py `with datalad code comes with a corresponding `tests/test_example.py`. |
| 58 |
Test helper functions assisting various general and DataLad specific assertions as well the construction of test directories and files can be found in ``datalad/tests/utils_pytest.py``. |
| 59 |
|
| 60 |
.. _decorators: |
| 61 |
|
| 62 |
Test annotations |
| 63 |
---------------- |
| 64 |
|
| 65 |
``datalad/tests/utils_pytest.py`` also defines test decorators. |
| 66 |
Some of those are used to annotate tests for various aspects to allow for easy sub-selection via environment variables. |
| 67 |
|
| 68 |
**Speed**: Please annotate tests that take a while to complete with following decorators |
| 69 |
|
| 70 |
* ``@slow`` if test runs over 10 seconds |
| 71 |
* ``@turtle`` if test runs over 120 seconds (those would not typically be ran on CIs) |
| 72 |
|
| 73 |
**Purpose**: Please further annotate tests with a special purpose specifically. As those tests also usually tend to be slower, use in conjunction with ``@slow`` or ``@turtle`` when slow. |
| 74 |
|
| 75 |
* ``@integration`` - tests verifying correct operation with external tools/services beyond git/git-annex |
| 76 |
* ``@usecase`` - represents some (user) use-case, and not necessarily a "unit-test" of functionality |
| 77 |
|
| 78 |
**Dysfunction**: If tests are not meant to be run on certain platforms or under certain conditions, ``@known_failure`` or ``@skip`` annotations can be used. Examples include: |
| 79 |
|
| 80 |
* ``@skip``, ``@skip_if_on_windows``, ``@skip_ssh``, ``@skip_wo_symlink_capability``, ``@skip_if_adjusted_branch``, ``@skip_if_no_network``, ``@skip_if_root`` |
| 81 |
* ``@knownfailure``, ``@known_failure_windows``, ``known_failure_githubci_win`` or ``known_failure_githubci_osx`` |
| 82 |
|
| 83 |
|
| 84 |
Migrating tests from nose to pytest |
| 85 |
=================================== |
| 86 |
|
| 87 |
DataLad's test suite has been migrated from `nose <https://nose.readthedocs.io/en/latest/>`_ to `pytest <https://docs.pytest.org/en/latest/contents.html>`_ in the `0.17.0 release <https://github.com/datalad/datalad/releases/tag/0.17.0>`_. |
| 88 |
This might be relevant for DataLad extensions that still use nose. |
| 89 |
|
| 90 |
For the time being, ``datalad.tests.utils`` keeps providing ``nose``-based utils, and ``datalad.__init__`` keeps providing nose-based fixtures to not break extensions that still use nose for testing. |
| 91 |
A migration to ``pytest`` is recommended, though. |
| 92 |
To perform a typical migration of a DataLad extension to use pytest instead of nose, go through the following list: |
| 93 |
|
| 94 |
* keep all the ``assert_*`` and ``ok_`` helpers, but import them from ``datalad.tests.utils_pytest`` instead |
| 95 |
* for ``@with_*`` and other decorators populating positional arguments, convert corresponding posarg to kwarg by adding ``=None`` |
| 96 |
* convert all generator-based parametric tests into direct invocations or, preferably, ``@pytest.mark.parametrized`` tests |
| 97 |
* address ``DeprecationWarnings`` in the code. Only where desired to test deprecation, add ``@pytest.mark.filterwarnings("ignore: BEGINNING OF WARNING")`` decorator to the test. |
| 98 |
|
| 99 |
For an example, see a "migrate to pytest" PR against ``datalad-deprecated``: `datalad/datalad-deprecated#51 <https://github.com/datalad/datalad-deprecated/pull/51>`_ . |