Debian Codegraph @

docs/internals/contributing/writing-code/unit-tests.txt

raw
1 ==========
2 Unit tests
3 ==========
4
5 Django comes with a test suite of its own, in the ``tests`` directory of the
6 code base. It's our policy to make sure all tests pass at all times.
7
8 We appreciate any and all contributions to the test suite!
9
10 The Django tests all use the testing infrastructure that ships with Django for
11 testing applications. See :doc:`/topics/testing/overview` for an explanation of
12 how to write new tests.
13
14 .. _running-unit-tests:
15
16 Running the unit tests
17 ======================
18
19 Quickstart
20 ----------
21
22 First, `fork Django on GitHub <https://github.com/django/django/fork>`__.
23
24 Second, create and activate a virtual environment. If you're not familiar with
25 how to do that, read our :doc:`contributing tutorial </intro/contributing>`.
26
27 Next, clone your fork, install some requirements, and run the tests:
28
29 .. console::
30
31 $ git clone https://github.com/YourGitHubName/django.git django-repo
32 $ cd django-repo/tests
33 $ python -m pip install -e ..
34 $ python -m pip install -r requirements/py3.txt
35 $ ./runtests.py
36
37 Installing the requirements will likely require some operating system packages
38 that your computer doesn't have installed. You can usually figure out which
39 package to install by doing a web search for the last line or so of the error
40 message. Try adding your operating system to the search query if needed.
41
42 If you have trouble installing the requirements, you can skip that step. See
43 :ref:`running-unit-tests-dependencies` for details on installing the optional
44 test dependencies. If you don't have an optional dependency installed, the
45 tests that require it will be skipped.
46
47 Running the tests requires a Django settings module that defines the databases
48 to use. To help you get started, Django provides and uses a sample settings
49 module that uses the SQLite database. See :ref:`running-unit-tests-settings` to
50 learn how to use a different settings module to run the tests with a different
51 database.
52
53 Having problems? See :ref:`troubleshooting-unit-tests` for some common issues.
54
55 Running tests using ``tox``
56 ---------------------------
57
58 `Tox <https://tox.wiki/>`_ is a tool for running tests in different virtual
59 environments. Django includes a basic ``tox.ini`` that automates some checks
60 that our build server performs on pull requests. To run the unit tests and
61 other checks (such as :ref:`import sorting <coding-style-imports>`, the
62 :ref:`documentation spelling checker <documentation-spelling-check>`, and
63 :ref:`code formatting <coding-style-python>`), install and run the ``tox``
64 command from any place in the Django source tree:
65
66 .. console::
67
68 $ python -m pip install tox
69 $ tox
70
71 By default, ``tox`` runs the test suite with the bundled test settings file for
72 SQLite, ``black``, ``blacken-docs``, ``flake8``, ``isort``, ``zizmor``, and the
73 documentation spelling checker. In addition to the system dependencies noted
74 elsewhere in this documentation, the command ``python3`` must be on your path
75 and linked to the appropriate version of Python. A list of default environments
76 can be seen as follows:
77
78 .. console::
79
80 $ tox -l
81 py3
82 black
83 blacken-docs
84 flake8>=3.7.0
85 docs
86 isort>=7.0.0
87 zizmor>=1.16.3
88
89 Testing other Python versions and database backends
90 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
91
92 In addition to the default environments, ``tox`` supports running unit tests
93 for other versions of Python and other database backends. Since Django's test
94 suite doesn't bundle a settings file for database backends other than SQLite,
95 however, you must :ref:`create and provide your own test settings
96 <running-unit-tests-settings>`. For example, to run the tests on Python 3.10
97 using PostgreSQL:
98
99 .. console::
100
101 $ tox -e py310-postgres -- --settings=my_postgres_settings
102
103 This command sets up a Python 3.10 virtual environment, installs Django's
104 test suite dependencies (including those for PostgreSQL), and calls
105 ``runtests.py`` with the supplied arguments (in this case,
106 ``--settings=my_postgres_settings``).
107
108 The remainder of this documentation shows commands for running tests without
109 ``tox``, however, any option passed to ``runtests.py`` can also be passed to
110 ``tox`` by prefixing the argument list with ``--``, as above.
111
112 ``Tox`` also respects the :envvar:`DJANGO_SETTINGS_MODULE` environment
113 variable, if set. For example, the following is equivalent to the command
114 above:
115
116 .. code-block:: console
117
118 $ DJANGO_SETTINGS_MODULE=my_postgres_settings tox -e py310-postgres
119
120 Windows users should use:
121
122 .. code-block:: doscon
123
124 ...\> set DJANGO_SETTINGS_MODULE=my_postgres_settings
125 ...\> tox -e py310-postgres
126
127 Running the JavaScript tests
128 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
129
130 Django includes a set of :ref:`JavaScript unit tests <javascript-tests>` for
131 functions in certain contrib apps. The JavaScript tests aren't run by default
132 using ``tox`` because they require ``Node.js`` to be installed and aren't
133 necessary for the majority of patches. To run the JavaScript tests using
134 ``tox``:
135
136 .. console::
137
138 $ tox -e javascript
139
140 This command runs ``npm install`` to ensure test requirements are up to
141 date and then runs ``npm test``.
142
143 Running tests using ``django-docker-box``
144 -----------------------------------------
145
146 `django-docker-box`_ allows you to run the Django's test suite across all
147 supported databases and python versions. See the `django-docker-box`_ project
148 page for installation and usage instructions.
149
150 .. _django-docker-box: https://github.com/django/django-docker-box/
151
152 .. _running-unit-tests-settings:
153
154 Using another ``settings`` module
155 ---------------------------------
156
157 The included settings module (``tests/test_sqlite.py``) allows you to run the
158 test suite using SQLite. If you want to run the tests using a different
159 database, you'll need to define your own settings file. Some tests, such as
160 those for ``contrib.postgres``, are specific to a particular database backend
161 and will be skipped if run with a different backend. Some tests are skipped or
162 expected failures on a particular database backend (see
163 ``DatabaseFeatures.django_test_skips`` and
164 ``DatabaseFeatures.django_test_expected_failures`` on each backend).
165
166 To run the tests with different settings, ensure that the module is on your
167 :envvar:`PYTHONPATH` and pass the module with ``--settings``.
168
169 The :setting:`DATABASES` setting in any test settings module needs to define
170 two databases:
171
172 * A ``default`` database. This database should use the backend that
173 you want to use for primary testing.
174
175 * A database with the alias ``other``. The ``other`` database is used to test
176 that queries can be directed to different databases. This database should use
177 the same backend as the ``default``, and it must have a different name.
178
179 If you're using a backend that isn't SQLite, you will need to provide other
180 details for each database:
181
182 * The :setting:`USER` option needs to specify an existing user account
183 for the database. That user needs permission to execute ``CREATE DATABASE``
184 so that the test database can be created.
185
186 * The :setting:`PASSWORD` option needs to provide the password for
187 the :setting:`USER` that has been specified.
188
189 Test databases get their names by prepending ``test_`` to the value of the
190 :setting:`NAME` settings for the databases defined in :setting:`DATABASES`.
191 These test databases are deleted when the tests are finished.
192
193 You will also need to ensure that your database uses UTF-8 as the default
194 character set. If your database server doesn't use UTF-8 as a default charset,
195 you will need to include a value for :setting:`CHARSET <TEST_CHARSET>` in the
196 test settings dictionary for the applicable database.
197
198 .. _runtests-specifying-labels:
199
200 Running only some of the tests
201 ------------------------------
202
203 Django's entire test suite takes a while to run, and running every single test
204 could be redundant if, say, you just added a test to Django that you want to
205 run quickly without running everything else. You can run a subset of the unit
206 tests by appending the names of the test modules to ``runtests.py`` on the
207 command line.
208
209 For example, if you'd like to run tests only for generic relations and
210 internationalization, type:
211
212 .. console::
213
214 $ ./runtests.py --settings=path.to.settings generic_relations i18n
215
216 How do you find out the names of individual tests? Look in ``tests/`` — each
217 directory name there is the name of a test.
218
219 If you want to run only a particular class of tests, you can specify a list of
220 paths to individual test classes. For example, to run the ``TranslationTests``
221 of the ``i18n`` module, type:
222
223 .. console::
224
225 $ ./runtests.py --settings=path.to.settings i18n.tests.TranslationTests
226
227 Going beyond that, you can specify an individual test method like this:
228
229 .. console::
230
231 $ ./runtests.py --settings=path.to.settings i18n.tests.TranslationTests.test_lazy_objects
232
233 You can run tests starting at a specified top-level module with ``--start-at``
234 option. For example:
235
236 .. console::
237
238 $ ./runtests.py --start-at=wsgi
239
240 You can also run tests starting after a specified top-level module with
241 ``--start-after`` option. For example:
242
243 .. console::
244
245 $ ./runtests.py --start-after=wsgi
246
247 Note that the ``--reverse`` option doesn't impact on ``--start-at`` or
248 ``--start-after`` options. Moreover these options cannot be used with test
249 labels.
250
251 .. _running-selenium-tests:
252
253 Running the Selenium tests
254 --------------------------
255
256 Some tests require Selenium and a web browser. To run these tests, you must
257 install the :pypi:`selenium` package and run the tests with the
258 ``--selenium=<BROWSERS>`` option. For example, if you have Firefox and Google
259 Chrome installed:
260
261 .. console::
262
263 $ ./runtests.py --selenium=firefox,chrome
264
265 See the `selenium.webdriver`_ package for the list of available browsers.
266
267 Specifying ``--selenium`` automatically sets ``--tags=selenium`` to run only
268 the tests that require selenium.
269
270 Some browsers (e.g. Chrome or Firefox) support headless testing, which can be
271 faster and more stable. Add the ``--headless`` option to enable this mode.
272
273 .. _selenium.webdriver: https://github.com/SeleniumHQ/selenium/tree/trunk/py/selenium/webdriver
274
275 .. _screenshot-tests:
276
277 Screenshot tests
278 ~~~~~~~~~~~~~~~~
279
280 For testing changes to the admin UI, the selenium tests can be run with the
281 ``--screenshots`` option enabled. Screenshots will be saved to the
282 ``tests/screenshots/`` directory.
283
284 To define when screenshots should be taken during a selenium test, the test
285 class must use the ``@django.test.selenium.screenshot_cases`` decorator with a
286 list of supported screenshot types (``"desktop_size"``, ``"mobile_size"``,
287 ``"small_screen_size"``, ``"rtl"``, ``"dark"``, and ``"high_contrast"``). It
288 can then call ``self.take_screenshot("unique-screenshot-name")`` at the desired
289 point to generate the screenshots. For example::
290
291 from django.test.selenium import SeleniumTestCase, screenshot_cases
292 from django.urls import reverse
293
294
295 class SeleniumTests(SeleniumTestCase):
296 @screenshot_cases(["desktop_size", "mobile_size", "rtl", "dark", "high_contrast"])
297 def test_login_button_centered(self):
298 self.selenium.get(self.live_server_url + reverse("admin:login"))
299 self.take_screenshot("login")
300 ...
301
302 This generates multiple screenshots of the login page - one for a desktop
303 screen, one for a mobile screen, one for right-to-left languages on desktop,
304 one for the dark mode on desktop, and one for high contrast mode on desktop
305 when using chrome.
306
307 .. versionchanged:: 5.1
308
309 The ``--screenshots`` option and ``@screenshot_cases`` decorator were
310 added.
311
312 .. _running-unit-tests-dependencies:
313
314 Running all the tests
315 ---------------------
316
317 If you want to run the full suite of tests, you'll need to install a number of
318 dependencies:
319
320 * :pypi:`aiosmtpd`
321 * :pypi:`argon2-cffi` 19.2.0+
322 * :pypi:`asgiref` 3.8.1+ (required)
323 * :pypi:`bcrypt`
324 * :pypi:`colorama` 0.4.6+
325 * :pypi:`docutils` 0.19+
326 * :pypi:`geoip2`
327 * :pypi:`Jinja2` 2.11+
328 * :pypi:`numpy`
329 * :pypi:`Pillow` 6.2.1+
330 * :pypi:`PyYAML`
331 * :pypi:`pywatchman`
332 * :pypi:`redis` 3.4+
333 * :pypi:`setuptools`
334 * :pypi:`pymemcache`, plus a `supported Python binding
335 <https://memcached.org/>`_
336 * `gettext <https://www.gnu.org/software/gettext/manual/gettext.html>`_
337 (:ref:`gettext_on_windows`)
338 * :pypi:`selenium` 4.8.0+
339 * :pypi:`sqlparse` 0.3.1+ (required)
340 * :pypi:`tblib` 1.5.0+
341
342 You can find these dependencies in `pip requirements files
343 <https://pip.pypa.io/en/latest/user_guide/#requirements-files>`_ inside the
344 ``tests/requirements`` directory of the Django source tree and install them
345 like so:
346
347 .. console::
348
349 $ python -m pip install -r tests/requirements/py3.txt
350
351 If you encounter an error during the installation, your system might be missing
352 a dependency for one or more of the Python packages. Consult the failing
353 package's documentation or search the web with the error message that you
354 encounter.
355
356 You can also install the database adapter(s) of your choice using
357 ``oracle.txt``, ``mysql.txt``, or ``postgres.txt``.
358
359 If you want to test the memcached or Redis cache backends, you'll also need to
360 define a :setting:`CACHES` setting that points at your memcached or Redis
361 instance respectively.
362
363 To run the GeoDjango tests, you will need to :doc:`set up a spatial database
364 and install the Geospatial libraries</ref/contrib/gis/install/index>`.
365
366 Each of these dependencies is optional. If you're missing any of them, the
367 associated tests will be skipped.
368
369 To run some of the autoreload tests, you'll need to install the
370 `Watchman <https://facebook.github.io/watchman/>`_ service.
371
372 Code coverage
373 -------------
374
375 Contributors are encouraged to run coverage on the test suite to identify areas
376 that need additional tests. The coverage tool installation and use is described
377 in :ref:`testing code coverage<topics-testing-code-coverage>`.
378
379 To run coverage on the Django test suite using the standard test settings:
380
381 .. console::
382
383 $ coverage run ./runtests.py --settings=test_sqlite
384
385 After running coverage, combine all coverage statistics by running:
386
387 .. console::
388
389 $ coverage combine
390
391 After that generate the html report by running:
392
393 .. console::
394
395 $ coverage html
396
397 When running coverage for the Django tests, the included ``.coveragerc``
398 settings file defines ``coverage_html`` as the output directory for the report
399 and also excludes several directories not relevant to the results
400 (test code or external code included in Django).
401
402 .. _contrib-apps:
403
404 Contrib apps
405 ============
406
407 Tests for contrib apps can be found in the :source:`tests/` directory, typically
408 under ``<app_name>_tests``. For example, tests for ``contrib.auth`` are located
409 in :source:`tests/auth_tests`.
410
411 .. _troubleshooting-unit-tests:
412
413 Troubleshooting
414 ===============
415
416 Test suite hangs or shows failures on ``main`` branch
417 -----------------------------------------------------
418
419 Ensure you have the latest point release of a :ref:`supported Python version
420 <faq-python-version-support>`, since there are often bugs in earlier versions
421 that may cause the test suite to fail or hang.
422
423 On **macOS** (High Sierra and newer versions), you might see this message
424 logged, after which the tests hang:
425
426 .. code-block:: pytb
427
428 objc[42074]: +[__NSPlaceholderDate initialize] may have been in progress in
429 another thread when fork() was called.
430
431 To avoid this set a ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment
432 variable, for example:
433
434 .. code-block:: shell
435
436 $ OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ./runtests.py
437
438 Or add ``export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES`` to your shell's
439 startup file (e.g. ``~/.profile``).
440
441 Many test failures with ``UnicodeEncodeError``
442 ----------------------------------------------
443
444 If the ``locales`` package is not installed, some tests will fail with a
445 ``UnicodeEncodeError``.
446
447 You can resolve this on Debian-based systems, for example, by running:
448
449 .. code-block:: console
450
451 $ apt-get install locales
452 $ dpkg-reconfigure locales
453
454 You can resolve this for macOS systems by configuring your shell's locale:
455
456 .. code-block:: console
457
458 $ export LANG="en_US.UTF-8"
459 $ export LC_ALL="en_US.UTF-8"
460
461 Run the ``locale`` command to confirm the change. Optionally, add those export
462 commands to your shell's startup file (e.g. ``~/.bashrc`` for Bash) to avoid
463 having to retype them.
464
465 Tests that only fail in combination
466 -----------------------------------
467
468 In case a test passes when run in isolation but fails within the whole suite,
469 we have some tools to help analyze the problem.
470
471 The ``--bisect`` option of ``runtests.py`` will run the failing test while
472 halving the test set it is run together with on each iteration, often making
473 it possible to identify a small number of tests that may be related to the
474 failure.
475
476 For example, suppose that the failing test that works on its own is
477 ``ModelTest.test_eq``, then using:
478
479 .. console::
480
481 $ ./runtests.py --bisect basic.tests.ModelTest.test_eq
482
483 will try to determine a test that interferes with the given one. First, the
484 test is run with the first half of the test suite. If a failure occurs, the
485 first half of the test suite is split in two groups and each group is then run
486 with the specified test. If there is no failure with the first half of the test
487 suite, the second half of the test suite is run with the specified test and
488 split appropriately as described earlier. The process repeats until the set of
489 failing tests is minimized.
490
491 The ``--pair`` option runs the given test alongside every other test from the
492 suite, letting you check if another test has side-effects that cause the
493 failure. So:
494
495 .. console::
496
497 $ ./runtests.py --pair basic.tests.ModelTest.test_eq
498
499 will pair ``test_eq`` with every test label.
500
501 With both ``--bisect`` and ``--pair``, if you already suspect which cases
502 might be responsible for the failure, you may limit tests to be cross-analyzed
503 by :ref:`specifying further test labels <runtests-specifying-labels>` after
504 the first one:
505
506 .. console::
507
508 $ ./runtests.py --pair basic.tests.ModelTest.test_eq queries transactions
509
510 You can also try running any set of tests in a random or reverse order using
511 the ``--shuffle`` and ``--reverse`` options. This can help verify that
512 executing tests in a different order does not cause any trouble:
513
514 .. console::
515
516 $ ./runtests.py basic --shuffle
517 $ ./runtests.py basic --reverse
518
519 Seeing the SQL queries run during a test
520 ----------------------------------------
521
522 If you wish to examine the SQL being run in failing tests, you can turn on
523 :ref:`SQL logging <django-db-logger>` using the ``--debug-sql`` option. If you
524 combine this with ``--verbosity=2``, all SQL queries will be output:
525
526 .. console::
527
528 $ ./runtests.py basic --debug-sql
529
530 Seeing the full traceback of a test failure
531 -------------------------------------------
532
533 By default tests are run in parallel with one process per core. When the tests
534 are run in parallel, however, you'll only see a truncated traceback for any
535 test failures. You can adjust this behavior with the ``--parallel`` option:
536
537 .. console::
538
539 $ ./runtests.py basic --parallel=1
540
541 You can also use the :envvar:`DJANGO_TEST_PROCESSES` environment variable for
542 this purpose.
543
544 Tips for writing tests
545 ======================
546
547 Isolating model registration
548 ----------------------------
549
550 To avoid polluting the global :attr:`~django.apps.apps` registry and prevent
551 unnecessary table creation, models defined in a test method should be bound to
552 a temporary ``Apps`` instance. To do this, use the
553 :func:`~django.test.utils.isolate_apps` decorator::
554
555 from django.db import models
556 from django.test import SimpleTestCase
557 from django.test.utils import isolate_apps
558
559
560 class TestModelDefinition(SimpleTestCase):
561 @isolate_apps("app_label")
562 def test_model_definition(self):
563 class TestModel(models.Model):
564 pass
565
566 ...
567
568 .. admonition:: Setting ``app_label``
569
570 Models defined in a test method with no explicit
571 :attr:`~django.db.models.Options.app_label` are automatically assigned the
572 label of the app in which their test class is located.
573
574 In order to make sure the models defined within the context of
575 :func:`~django.test.utils.isolate_apps` instances are correctly
576 installed, you should pass the set of targeted ``app_label`` as arguments:
577
578 .. code-block:: python
579 :caption: ``tests/app_label/tests.py``
580
581 from django.db import models
582 from django.test import SimpleTestCase
583 from django.test.utils import isolate_apps
584
585
586 class TestModelDefinition(SimpleTestCase):
587 @isolate_apps("app_label", "other_app_label")
588 def test_model_definition(self):
589 # This model automatically receives app_label='app_label'
590 class TestModel(models.Model):
591 pass
592
593 class OtherAppModel(models.Model):
594 class Meta:
595 app_label = "other_app_label"
596
597 ...