| 1 |
============================ |
| 2 |
Performance and optimization |
| 3 |
============================ |
| 4 |
|
| 5 |
This document provides an overview of techniques and tools that can help get |
| 6 |
your Django code running more efficiently - faster, and using fewer system |
| 7 |
resources. |
| 8 |
|
| 9 |
Introduction |
| 10 |
============ |
| 11 |
|
| 12 |
Generally one's first concern is to write code that *works*, whose logic |
| 13 |
functions as required to produce the expected output. Sometimes, however, this |
| 14 |
will not be enough to make the code work as *efficiently* as one would like. |
| 15 |
|
| 16 |
In this case, what's needed is something - and in practice, often a collection |
| 17 |
of things - to improve the code's performance without, or only minimally, |
| 18 |
affecting its behavior. |
| 19 |
|
| 20 |
General approaches |
| 21 |
================== |
| 22 |
|
| 23 |
What are you optimizing *for*? |
| 24 |
------------------------------ |
| 25 |
|
| 26 |
It's important to have a clear idea what you mean by 'performance'. There is |
| 27 |
not just one metric of it. |
| 28 |
|
| 29 |
Improved speed might be the most obvious aim for a program, but sometimes other |
| 30 |
performance improvements might be sought, such as lower memory consumption or |
| 31 |
fewer demands on the database or network. |
| 32 |
|
| 33 |
Improvements in one area will often bring about improved performance in |
| 34 |
another, but not always; sometimes one can even be at the expense of another. |
| 35 |
For example, an improvement in a program's speed might cause it to use more |
| 36 |
memory. Even worse, it can be self-defeating - if the speed improvement is so |
| 37 |
memory-hungry that the system starts to run out of memory, you'll have done |
| 38 |
more harm than good. |
| 39 |
|
| 40 |
There are other trade-offs to bear in mind. Your own time is a valuable |
| 41 |
resource, more precious than CPU time. Some improvements might be too difficult |
| 42 |
to be worth implementing, or might affect the portability or maintainability of |
| 43 |
the code. Not all performance improvements are worth the effort. |
| 44 |
|
| 45 |
So, you need to know what performance improvements you are aiming for, and you |
| 46 |
also need to know that you have a good reason for aiming in that direction - |
| 47 |
and for that you need: |
| 48 |
|
| 49 |
Performance benchmarking |
| 50 |
------------------------ |
| 51 |
|
| 52 |
It's no good just guessing or assuming where the inefficiencies lie in your |
| 53 |
code. |
| 54 |
|
| 55 |
Django tools |
| 56 |
~~~~~~~~~~~~ |
| 57 |
|
| 58 |
:pypi:`django-debug-toolbar` is a very handy tool that provides insights into |
| 59 |
what your code is doing and how much time it spends doing it. In particular it |
| 60 |
can show you all the SQL queries your page is generating, and how long each one |
| 61 |
has taken. |
| 62 |
|
| 63 |
Third-party panels are also available for the toolbar, that can (for example) |
| 64 |
report on cache performance and template rendering times. |
| 65 |
|
| 66 |
Third-party services |
| 67 |
~~~~~~~~~~~~~~~~~~~~ |
| 68 |
|
| 69 |
There are a number of free services that will analyze and report on the |
| 70 |
performance of your site's pages from the perspective of a remote HTTP client, |
| 71 |
in effect simulating the experience of an actual user. |
| 72 |
|
| 73 |
These can't report on the internals of your code, but can provide a useful |
| 74 |
insight into your site's overall performance, including aspects that can't be |
| 75 |
adequately measured from within Django environment. |
| 76 |
|
| 77 |
There are also several paid-for services that perform a similar analysis, |
| 78 |
including some that are Django-aware and can integrate with your codebase to |
| 79 |
profile its performance far more comprehensively. |
| 80 |
|
| 81 |
Get things right from the start |
| 82 |
------------------------------- |
| 83 |
|
| 84 |
Some work in optimization involves tackling performance shortcomings, but some |
| 85 |
of the work can be built-in to what you'd do anyway, as part of the good |
| 86 |
practices you should adopt even before you start thinking about improving |
| 87 |
performance. |
| 88 |
|
| 89 |
In this respect Python is an excellent language to work with, because solutions |
| 90 |
that look elegant and feel right usually are the best performing ones. As with |
| 91 |
most skills, learning what "looks right" takes practice, but one of the most |
| 92 |
useful guidelines is: |
| 93 |
|
| 94 |
Work at the appropriate level |
| 95 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 96 |
|
| 97 |
Django offers many different ways of approaching things, but just because it's |
| 98 |
possible to do something in a certain way doesn't mean that it's the most |
| 99 |
appropriate way to do it. For example, you might find that you could calculate |
| 100 |
the same thing - the number of items in a collection, perhaps - in a |
| 101 |
``QuerySet``, in Python, or in a template. |
| 102 |
|
| 103 |
However, it will almost always be faster to do this work at lower rather than |
| 104 |
higher levels. At higher levels the system has to deal with objects through |
| 105 |
multiple levels of abstraction and layers of machinery. |
| 106 |
|
| 107 |
That is, the database can typically do things faster than Python can, which can |
| 108 |
do them faster than the template language can:: |
| 109 |
|
| 110 |
# QuerySet operation on the database |
| 111 |
# fast, because that's what databases are good at |
| 112 |
my_bicycles.count() |
| 113 |
|
| 114 |
# counting Python objects |
| 115 |
# slower, because it requires a database query anyway, and processing |
| 116 |
# of the Python objects |
| 117 |
len(my_bicycles) |
| 118 |
|
| 119 |
.. code-block:: html+django |
| 120 |
|
| 121 |
<!-- |
| 122 |
Django template filter |
| 123 |
slower still, because it will have to count them in Python anyway, |
| 124 |
and because of template language overheads |
| 125 |
--> |
| 126 |
{{ my_bicycles|length }} |
| 127 |
|
| 128 |
Generally speaking, the most appropriate level for the job is the lowest-level |
| 129 |
one that it is comfortable to code for. |
| 130 |
|
| 131 |
.. note:: |
| 132 |
|
| 133 |
The example above is merely illustrative. |
| 134 |
|
| 135 |
Firstly, in a real-life case you need to consider what is happening before |
| 136 |
and after your count to work out what's an optimal way of doing it *in that |
| 137 |
particular context*. The database optimization document describes :ref:`a |
| 138 |
case where counting in the template would be better |
| 139 |
<overuse_of_count_and_exists>`. |
| 140 |
|
| 141 |
Secondly, there are other options to consider: in a real-life case, ``{{ |
| 142 |
my_bicycles.count }}``, which invokes the ``QuerySet`` ``count()`` method |
| 143 |
directly from the template, might be the most appropriate choice. |
| 144 |
|
| 145 |
Caching |
| 146 |
======= |
| 147 |
|
| 148 |
Often it is expensive (that is, resource-hungry and slow) to compute a value, |
| 149 |
so there can be huge benefit in saving the value to a quickly accessible cache, |
| 150 |
ready for the next time it's required. |
| 151 |
|
| 152 |
It's a sufficiently significant and powerful technique that Django includes a |
| 153 |
comprehensive caching framework, as well as other smaller pieces of caching |
| 154 |
functionality. |
| 155 |
|
| 156 |
:doc:`The caching framework </topics/cache>` |
| 157 |
-------------------------------------------- |
| 158 |
|
| 159 |
Django's :doc:`caching framework </topics/cache>` offers very significant |
| 160 |
opportunities for performance gains, by saving dynamic content so that it |
| 161 |
doesn't need to be calculated for each request. |
| 162 |
|
| 163 |
For convenience, Django offers different levels of cache granularity: you can |
| 164 |
cache the output of specific views, or only the pieces that are difficult to |
| 165 |
produce, or even an entire site. |
| 166 |
|
| 167 |
Implementing caching should not be regarded as an alternative to improving code |
| 168 |
that's performing poorly because it has been written badly. It's one of the |
| 169 |
final steps toward producing well-performing code, not a shortcut. |
| 170 |
|
| 171 |
:class:`~django.utils.functional.cached_property` |
| 172 |
------------------------------------------------- |
| 173 |
|
| 174 |
It's common to have to call a class instance's method more than once. If |
| 175 |
that function is expensive, then doing so can be wasteful. |
| 176 |
|
| 177 |
Using the :class:`~django.utils.functional.cached_property` decorator saves the |
| 178 |
value returned by a property; the next time the function is called on that |
| 179 |
instance, it will return the saved value rather than re-computing it. Note that |
| 180 |
this only works on methods that take ``self`` as their only argument and that |
| 181 |
it changes the method to a property. |
| 182 |
|
| 183 |
Certain Django components also have their own caching functionality; these are |
| 184 |
discussed below in the sections related to those components. |
| 185 |
|
| 186 |
Understanding laziness |
| 187 |
====================== |
| 188 |
|
| 189 |
*Laziness* is a strategy complementary to caching. Caching avoids |
| 190 |
recomputation by saving results; laziness delays computation until it's |
| 191 |
actually required. |
| 192 |
|
| 193 |
Laziness allows us to refer to things before they are instantiated, or even |
| 194 |
before it's possible to instantiate them. This has numerous uses. |
| 195 |
|
| 196 |
For example, :ref:`lazy translation <lazy-translations>` can be used before the |
| 197 |
target language is even known, because it doesn't take place until the |
| 198 |
translated string is actually required, such as in a rendered template. |
| 199 |
|
| 200 |
Laziness is also a way to save effort by trying to avoid work in the first |
| 201 |
place. That is, one aspect of laziness is not doing anything until it has to be |
| 202 |
done, because it may not turn out to be necessary after all. Laziness can |
| 203 |
therefore have performance implications, and the more expensive the work |
| 204 |
concerned, the more there is to gain through laziness. |
| 205 |
|
| 206 |
Python provides a number of tools for lazy evaluation, particularly through the |
| 207 |
:py:term:`generator` and :py:term:`generator expression` constructs. It's worth |
| 208 |
reading up on laziness in Python to discover opportunities for making use of |
| 209 |
lazy patterns in your code. |
| 210 |
|
| 211 |
Laziness in Django |
| 212 |
------------------ |
| 213 |
|
| 214 |
Django is itself quite lazy. A good example of this can be found in the |
| 215 |
evaluation of a ``QuerySet``. :ref:`QuerySets are lazy <querysets-are-lazy>`. |
| 216 |
Thus a ``QuerySet`` can be created, passed around and combined with other |
| 217 |
``QuerySet`` instances, without actually incurring any trips to the database |
| 218 |
to fetch the items it describes. What gets passed around is the ``QuerySet`` |
| 219 |
object, not the collection of items that - eventually - will be required from |
| 220 |
the database. |
| 221 |
|
| 222 |
On the other hand, :ref:`certain operations will force the evaluation of a |
| 223 |
QuerySet <when-querysets-are-evaluated>`. Avoiding the premature evaluation of |
| 224 |
a ``QuerySet`` can save making an expensive and unnecessary trip to the |
| 225 |
database. |
| 226 |
|
| 227 |
Django also offers a :meth:`~django.utils.functional.keep_lazy` decorator. |
| 228 |
This allows a function that has been called with a lazy argument to behave |
| 229 |
lazily itself, only being evaluated when it needs to be. Thus the lazy argument |
| 230 |
- which could be an expensive one - will not be called upon for evaluation |
| 231 |
until it's strictly required. |
| 232 |
|
| 233 |
Databases |
| 234 |
========= |
| 235 |
|
| 236 |
Database optimization |
| 237 |
--------------------- |
| 238 |
|
| 239 |
Django's database layer provides various ways to help developers get the best |
| 240 |
performance from their databases. The :doc:`database optimization documentation |
| 241 |
</topics/db/optimization>` gathers together links to the relevant |
| 242 |
documentation and adds various tips that outline the steps to take when |
| 243 |
attempting to optimize your database usage. |
| 244 |
|
| 245 |
Other database-related tips |
| 246 |
--------------------------- |
| 247 |
|
| 248 |
Enabling :ref:`persistent-database-connections` can speed up connections to the |
| 249 |
database accounts for a significant part of the request processing time. |
| 250 |
|
| 251 |
This helps a lot on virtualized hosts with limited network performance, for example. |
| 252 |
|
| 253 |
HTTP performance |
| 254 |
================ |
| 255 |
|
| 256 |
Middleware |
| 257 |
---------- |
| 258 |
|
| 259 |
Django comes with a few helpful pieces of :doc:`middleware </ref/middleware>` |
| 260 |
that can help optimize your site's performance. They include: |
| 261 |
|
| 262 |
:class:`~django.middleware.http.ConditionalGetMiddleware` |
| 263 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 264 |
|
| 265 |
Adds support for modern browsers to conditionally GET responses based on the |
| 266 |
``ETag`` and ``Last-Modified`` headers. It also calculates and sets an ETag if |
| 267 |
needed. |
| 268 |
|
| 269 |
:class:`~django.middleware.gzip.GZipMiddleware` |
| 270 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 271 |
|
| 272 |
Compresses responses for all modern browsers, saving bandwidth and transfer |
| 273 |
time. Note that GZipMiddleware is currently considered a security risk, and is |
| 274 |
vulnerable to attacks that nullify the protection provided by TLS/SSL. See the |
| 275 |
warning in :class:`~django.middleware.gzip.GZipMiddleware` for more information. |
| 276 |
|
| 277 |
Sessions |
| 278 |
-------- |
| 279 |
|
| 280 |
Using cached sessions |
| 281 |
~~~~~~~~~~~~~~~~~~~~~ |
| 282 |
|
| 283 |
:ref:`Using cached sessions <cached-sessions-backend>` may be a way to increase |
| 284 |
performance by eliminating the need to load session data from a slower storage |
| 285 |
source like the database and instead storing frequently used session data in |
| 286 |
memory. |
| 287 |
|
| 288 |
Static files |
| 289 |
------------ |
| 290 |
|
| 291 |
Static files, which by definition are not dynamic, make an excellent target for |
| 292 |
optimization gains. |
| 293 |
|
| 294 |
:class:`~django.contrib.staticfiles.storage.ManifestStaticFilesStorage` |
| 295 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 296 |
|
| 297 |
By taking advantage of web browsers' caching abilities, you can |
| 298 |
eliminate network hits entirely for a given file after the initial download. |
| 299 |
|
| 300 |
:class:`~django.contrib.staticfiles.storage.ManifestStaticFilesStorage` appends a |
| 301 |
content-dependent tag to the filenames of :doc:`static files |
| 302 |
</ref/contrib/staticfiles>` to make it safe for browsers to cache them |
| 303 |
long-term without missing future changes - when a file changes, so will the |
| 304 |
tag, so browsers will reload the asset automatically. |
| 305 |
|
| 306 |
"Minification" |
| 307 |
~~~~~~~~~~~~~~ |
| 308 |
|
| 309 |
Several third-party Django tools and packages provide the ability to "minify" |
| 310 |
HTML, CSS, and JavaScript. They remove unnecessary whitespace, newlines, and |
| 311 |
comments, and shorten variable names, and thus reduce the size of the documents |
| 312 |
that your site publishes. |
| 313 |
|
| 314 |
Template performance |
| 315 |
==================== |
| 316 |
|
| 317 |
Note that: |
| 318 |
|
| 319 |
* using ``{% block %}`` is faster than using ``{% include %}`` |
| 320 |
* heavily-fragmented templates, assembled from many small pieces, can affect |
| 321 |
performance |
| 322 |
|
| 323 |
The cached template loader |
| 324 |
-------------------------- |
| 325 |
|
| 326 |
Enabling the :class:`cached template loader |
| 327 |
<django.template.loaders.cached.Loader>` often improves performance |
| 328 |
drastically, as it avoids compiling each template every time it needs to be |
| 329 |
rendered. |
| 330 |
|
| 331 |
Using different versions of available software |
| 332 |
============================================== |
| 333 |
|
| 334 |
It can sometimes be worth checking whether different and better-performing |
| 335 |
versions of the software that you're using are available. |
| 336 |
|
| 337 |
These techniques are targeted at more advanced users who want to push the |
| 338 |
boundaries of performance of an already well-optimized Django site. |
| 339 |
|
| 340 |
However, they are not magic solutions to performance problems, and they're |
| 341 |
unlikely to bring better than marginal gains to sites that don't already do the |
| 342 |
more basic things the right way. |
| 343 |
|
| 344 |
.. note:: |
| 345 |
|
| 346 |
It's worth repeating: **reaching for alternatives to software you're |
| 347 |
already using is never the first answer to performance problems**. When |
| 348 |
you reach this level of optimization, you need a formal benchmarking |
| 349 |
solution. |
| 350 |
|
| 351 |
Newer is often - but not always - better |
| 352 |
---------------------------------------- |
| 353 |
|
| 354 |
It's fairly rare for a new release of well-maintained software to be less |
| 355 |
efficient, but the maintainers can't anticipate every possible use-case - so |
| 356 |
while being aware that newer versions are likely to perform better, don't |
| 357 |
assume that they always will. |
| 358 |
|
| 359 |
This is true of Django itself. Successive releases have offered a number of |
| 360 |
improvements across the system, but you should still check the real-world |
| 361 |
performance of your application, because in some cases you may find that |
| 362 |
changes mean it performs worse rather than better. |
| 363 |
|
| 364 |
Newer versions of Python, and also of Python packages, will often perform |
| 365 |
better too - but measure, rather than assume. |
| 366 |
|
| 367 |
.. note:: |
| 368 |
|
| 369 |
Unless you've encountered an unusual performance problem in a particular |
| 370 |
version, you'll generally find better features, reliability, and security |
| 371 |
in a new release and that these benefits are far more significant than any |
| 372 |
performance you might win or lose. |
| 373 |
|
| 374 |
Alternatives to Django's template language |
| 375 |
------------------------------------------ |
| 376 |
|
| 377 |
For nearly all cases, Django's built-in template language is perfectly |
| 378 |
adequate. However, if the bottlenecks in your Django project seem to lie in the |
| 379 |
template system and you have exhausted other opportunities to remedy this, a |
| 380 |
third-party alternative may be the answer. |
| 381 |
|
| 382 |
Jinja2_ can offer performance improvements, particularly when it comes to |
| 383 |
speed. |
| 384 |
|
| 385 |
Alternative template systems vary in the extent to which they share Django's |
| 386 |
templating language. |
| 387 |
|
| 388 |
.. note:: |
| 389 |
|
| 390 |
*If* you experience performance issues in templates, the first thing to do |
| 391 |
is to understand exactly why. Using an alternative template system may |
| 392 |
prove faster, but the same gains may also be available without going to |
| 393 |
that trouble - for example, expensive processing and logic in your |
| 394 |
templates could be done more efficiently in your views. |
| 395 |
|
| 396 |
Alternative software implementations |
| 397 |
------------------------------------ |
| 398 |
|
| 399 |
It may be worth checking whether Python software you're using has been |
| 400 |
provided in a different implementation that can execute the same code faster. |
| 401 |
|
| 402 |
However: most performance problems in well-written Django sites aren't at the |
| 403 |
Python execution level, but rather in inefficient database querying, caching, |
| 404 |
and templates. If you're relying on poorly-written Python code, your |
| 405 |
performance problems are unlikely to be solved by having it execute faster. |
| 406 |
|
| 407 |
Using an alternative implementation may introduce compatibility, deployment, |
| 408 |
portability, or maintenance issues. It goes without saying that before adopting |
| 409 |
a non-standard implementation you should ensure it provides sufficient |
| 410 |
performance gains for your application to outweigh the potential risks. |
| 411 |
|
| 412 |
With these caveats in mind, you should be aware of: |
| 413 |
|
| 414 |
`PyPy <https://www.pypy.org/>`_ |
| 415 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 416 |
|
| 417 |
`PyPy <https://www.pypy.org/>`_ is an implementation of Python in Python itself |
| 418 |
(the 'standard' Python implementation is in C). PyPy can offer substantial |
| 419 |
performance gains, typically for heavyweight applications. |
| 420 |
|
| 421 |
A key aim of the PyPy project is `compatibility |
| 422 |
<https://www.pypy.org/compat.html>`_ with existing Python APIs and libraries. |
| 423 |
Django is compatible, but you will need to check the compatibility of other |
| 424 |
libraries you rely on. |
| 425 |
|
| 426 |
C implementations of Python libraries |
| 427 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 428 |
|
| 429 |
Some Python libraries are also implemented in C, and can be much faster. They |
| 430 |
aim to offer the same APIs. Note that compatibility issues and behavior |
| 431 |
differences are not unknown (and not always immediately evident). |
| 432 |
|
| 433 |
.. _Jinja2: https://jinja.palletsprojects.com/ |