Debian Codegraph @

docs/topics/performance.txt

raw
1 ============================
2 Performance and optimization
3 ============================
4
5 This document provides an overview of techniques and tools that can help get
6 your Django code running more efficiently - faster, and using fewer system
7 resources.
8
9 Introduction
10 ============
11
12 Generally one's first concern is to write code that *works*, whose logic
13 functions as required to produce the expected output. Sometimes, however, this
14 will not be enough to make the code work as *efficiently* as one would like.
15
16 In this case, what's needed is something - and in practice, often a collection
17 of things - to improve the code's performance without, or only minimally,
18 affecting its behavior.
19
20 General approaches
21 ==================
22
23 What are you optimizing *for*?
24 ------------------------------
25
26 It's important to have a clear idea what you mean by 'performance'. There is
27 not just one metric of it.
28
29 Improved speed might be the most obvious aim for a program, but sometimes other
30 performance improvements might be sought, such as lower memory consumption or
31 fewer demands on the database or network.
32
33 Improvements in one area will often bring about improved performance in
34 another, but not always; sometimes one can even be at the expense of another.
35 For example, an improvement in a program's speed might cause it to use more
36 memory. Even worse, it can be self-defeating - if the speed improvement is so
37 memory-hungry that the system starts to run out of memory, you'll have done
38 more harm than good.
39
40 There are other trade-offs to bear in mind. Your own time is a valuable
41 resource, more precious than CPU time. Some improvements might be too difficult
42 to be worth implementing, or might affect the portability or maintainability of
43 the code. Not all performance improvements are worth the effort.
44
45 So, you need to know what performance improvements you are aiming for, and you
46 also need to know that you have a good reason for aiming in that direction -
47 and for that you need:
48
49 Performance benchmarking
50 ------------------------
51
52 It's no good just guessing or assuming where the inefficiencies lie in your
53 code.
54
55 Django tools
56 ~~~~~~~~~~~~
57
58 :pypi:`django-debug-toolbar` is a very handy tool that provides insights into
59 what your code is doing and how much time it spends doing it. In particular it
60 can show you all the SQL queries your page is generating, and how long each one
61 has taken.
62
63 Third-party panels are also available for the toolbar, that can (for example)
64 report on cache performance and template rendering times.
65
66 Third-party services
67 ~~~~~~~~~~~~~~~~~~~~
68
69 There are a number of free services that will analyze and report on the
70 performance of your site's pages from the perspective of a remote HTTP client,
71 in effect simulating the experience of an actual user.
72
73 These can't report on the internals of your code, but can provide a useful
74 insight into your site's overall performance, including aspects that can't be
75 adequately measured from within Django environment.
76
77 There are also several paid-for services that perform a similar analysis,
78 including some that are Django-aware and can integrate with your codebase to
79 profile its performance far more comprehensively.
80
81 Get things right from the start
82 -------------------------------
83
84 Some work in optimization involves tackling performance shortcomings, but some
85 of the work can be built-in to what you'd do anyway, as part of the good
86 practices you should adopt even before you start thinking about improving
87 performance.
88
89 In this respect Python is an excellent language to work with, because solutions
90 that look elegant and feel right usually are the best performing ones. As with
91 most skills, learning what "looks right" takes practice, but one of the most
92 useful guidelines is:
93
94 Work at the appropriate level
95 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
96
97 Django offers many different ways of approaching things, but just because it's
98 possible to do something in a certain way doesn't mean that it's the most
99 appropriate way to do it. For example, you might find that you could calculate
100 the same thing - the number of items in a collection, perhaps - in a
101 ``QuerySet``, in Python, or in a template.
102
103 However, it will almost always be faster to do this work at lower rather than
104 higher levels. At higher levels the system has to deal with objects through
105 multiple levels of abstraction and layers of machinery.
106
107 That is, the database can typically do things faster than Python can, which can
108 do them faster than the template language can::
109
110 # QuerySet operation on the database
111 # fast, because that's what databases are good at
112 my_bicycles.count()
113
114 # counting Python objects
115 # slower, because it requires a database query anyway, and processing
116 # of the Python objects
117 len(my_bicycles)
118
119 .. code-block:: html+django
120
121 <!--
122 Django template filter
123 slower still, because it will have to count them in Python anyway,
124 and because of template language overheads
125 -->
126 {{ my_bicycles|length }}
127
128 Generally speaking, the most appropriate level for the job is the lowest-level
129 one that it is comfortable to code for.
130
131 .. note::
132
133 The example above is merely illustrative.
134
135 Firstly, in a real-life case you need to consider what is happening before
136 and after your count to work out what's an optimal way of doing it *in that
137 particular context*. The database optimization document describes :ref:`a
138 case where counting in the template would be better
139 <overuse_of_count_and_exists>`.
140
141 Secondly, there are other options to consider: in a real-life case, ``{{
142 my_bicycles.count }}``, which invokes the ``QuerySet`` ``count()`` method
143 directly from the template, might be the most appropriate choice.
144
145 Caching
146 =======
147
148 Often it is expensive (that is, resource-hungry and slow) to compute a value,
149 so there can be huge benefit in saving the value to a quickly accessible cache,
150 ready for the next time it's required.
151
152 It's a sufficiently significant and powerful technique that Django includes a
153 comprehensive caching framework, as well as other smaller pieces of caching
154 functionality.
155
156 :doc:`The caching framework </topics/cache>`
157 --------------------------------------------
158
159 Django's :doc:`caching framework </topics/cache>` offers very significant
160 opportunities for performance gains, by saving dynamic content so that it
161 doesn't need to be calculated for each request.
162
163 For convenience, Django offers different levels of cache granularity: you can
164 cache the output of specific views, or only the pieces that are difficult to
165 produce, or even an entire site.
166
167 Implementing caching should not be regarded as an alternative to improving code
168 that's performing poorly because it has been written badly. It's one of the
169 final steps toward producing well-performing code, not a shortcut.
170
171 :class:`~django.utils.functional.cached_property`
172 -------------------------------------------------
173
174 It's common to have to call a class instance's method more than once. If
175 that function is expensive, then doing so can be wasteful.
176
177 Using the :class:`~django.utils.functional.cached_property` decorator saves the
178 value returned by a property; the next time the function is called on that
179 instance, it will return the saved value rather than re-computing it. Note that
180 this only works on methods that take ``self`` as their only argument and that
181 it changes the method to a property.
182
183 Certain Django components also have their own caching functionality; these are
184 discussed below in the sections related to those components.
185
186 Understanding laziness
187 ======================
188
189 *Laziness* is a strategy complementary to caching. Caching avoids
190 recomputation by saving results; laziness delays computation until it's
191 actually required.
192
193 Laziness allows us to refer to things before they are instantiated, or even
194 before it's possible to instantiate them. This has numerous uses.
195
196 For example, :ref:`lazy translation <lazy-translations>` can be used before the
197 target language is even known, because it doesn't take place until the
198 translated string is actually required, such as in a rendered template.
199
200 Laziness is also a way to save effort by trying to avoid work in the first
201 place. That is, one aspect of laziness is not doing anything until it has to be
202 done, because it may not turn out to be necessary after all. Laziness can
203 therefore have performance implications, and the more expensive the work
204 concerned, the more there is to gain through laziness.
205
206 Python provides a number of tools for lazy evaluation, particularly through the
207 :py:term:`generator` and :py:term:`generator expression` constructs. It's worth
208 reading up on laziness in Python to discover opportunities for making use of
209 lazy patterns in your code.
210
211 Laziness in Django
212 ------------------
213
214 Django is itself quite lazy. A good example of this can be found in the
215 evaluation of a ``QuerySet``. :ref:`QuerySets are lazy <querysets-are-lazy>`.
216 Thus a ``QuerySet`` can be created, passed around and combined with other
217 ``QuerySet`` instances, without actually incurring any trips to the database
218 to fetch the items it describes. What gets passed around is the ``QuerySet``
219 object, not the collection of items that - eventually - will be required from
220 the database.
221
222 On the other hand, :ref:`certain operations will force the evaluation of a
223 QuerySet <when-querysets-are-evaluated>`. Avoiding the premature evaluation of
224 a ``QuerySet`` can save making an expensive and unnecessary trip to the
225 database.
226
227 Django also offers a :meth:`~django.utils.functional.keep_lazy` decorator.
228 This allows a function that has been called with a lazy argument to behave
229 lazily itself, only being evaluated when it needs to be. Thus the lazy argument
230 - which could be an expensive one - will not be called upon for evaluation
231 until it's strictly required.
232
233 Databases
234 =========
235
236 Database optimization
237 ---------------------
238
239 Django's database layer provides various ways to help developers get the best
240 performance from their databases. The :doc:`database optimization documentation
241 </topics/db/optimization>` gathers together links to the relevant
242 documentation and adds various tips that outline the steps to take when
243 attempting to optimize your database usage.
244
245 Other database-related tips
246 ---------------------------
247
248 Enabling :ref:`persistent-database-connections` can speed up connections to the
249 database accounts for a significant part of the request processing time.
250
251 This helps a lot on virtualized hosts with limited network performance, for example.
252
253 HTTP performance
254 ================
255
256 Middleware
257 ----------
258
259 Django comes with a few helpful pieces of :doc:`middleware </ref/middleware>`
260 that can help optimize your site's performance. They include:
261
262 :class:`~django.middleware.http.ConditionalGetMiddleware`
263 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
264
265 Adds support for modern browsers to conditionally GET responses based on the
266 ``ETag`` and ``Last-Modified`` headers. It also calculates and sets an ETag if
267 needed.
268
269 :class:`~django.middleware.gzip.GZipMiddleware`
270 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
271
272 Compresses responses for all modern browsers, saving bandwidth and transfer
273 time. Note that GZipMiddleware is currently considered a security risk, and is
274 vulnerable to attacks that nullify the protection provided by TLS/SSL. See the
275 warning in :class:`~django.middleware.gzip.GZipMiddleware` for more information.
276
277 Sessions
278 --------
279
280 Using cached sessions
281 ~~~~~~~~~~~~~~~~~~~~~
282
283 :ref:`Using cached sessions <cached-sessions-backend>` may be a way to increase
284 performance by eliminating the need to load session data from a slower storage
285 source like the database and instead storing frequently used session data in
286 memory.
287
288 Static files
289 ------------
290
291 Static files, which by definition are not dynamic, make an excellent target for
292 optimization gains.
293
294 :class:`~django.contrib.staticfiles.storage.ManifestStaticFilesStorage`
295 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
296
297 By taking advantage of web browsers' caching abilities, you can
298 eliminate network hits entirely for a given file after the initial download.
299
300 :class:`~django.contrib.staticfiles.storage.ManifestStaticFilesStorage` appends a
301 content-dependent tag to the filenames of :doc:`static files
302 </ref/contrib/staticfiles>` to make it safe for browsers to cache them
303 long-term without missing future changes - when a file changes, so will the
304 tag, so browsers will reload the asset automatically.
305
306 "Minification"
307 ~~~~~~~~~~~~~~
308
309 Several third-party Django tools and packages provide the ability to "minify"
310 HTML, CSS, and JavaScript. They remove unnecessary whitespace, newlines, and
311 comments, and shorten variable names, and thus reduce the size of the documents
312 that your site publishes.
313
314 Template performance
315 ====================
316
317 Note that:
318
319 * using ``{% block %}`` is faster than using ``{% include %}``
320 * heavily-fragmented templates, assembled from many small pieces, can affect
321 performance
322
323 The cached template loader
324 --------------------------
325
326 Enabling the :class:`cached template loader
327 <django.template.loaders.cached.Loader>` often improves performance
328 drastically, as it avoids compiling each template every time it needs to be
329 rendered.
330
331 Using different versions of available software
332 ==============================================
333
334 It can sometimes be worth checking whether different and better-performing
335 versions of the software that you're using are available.
336
337 These techniques are targeted at more advanced users who want to push the
338 boundaries of performance of an already well-optimized Django site.
339
340 However, they are not magic solutions to performance problems, and they're
341 unlikely to bring better than marginal gains to sites that don't already do the
342 more basic things the right way.
343
344 .. note::
345
346 It's worth repeating: **reaching for alternatives to software you're
347 already using is never the first answer to performance problems**. When
348 you reach this level of optimization, you need a formal benchmarking
349 solution.
350
351 Newer is often - but not always - better
352 ----------------------------------------
353
354 It's fairly rare for a new release of well-maintained software to be less
355 efficient, but the maintainers can't anticipate every possible use-case - so
356 while being aware that newer versions are likely to perform better, don't
357 assume that they always will.
358
359 This is true of Django itself. Successive releases have offered a number of
360 improvements across the system, but you should still check the real-world
361 performance of your application, because in some cases you may find that
362 changes mean it performs worse rather than better.
363
364 Newer versions of Python, and also of Python packages, will often perform
365 better too - but measure, rather than assume.
366
367 .. note::
368
369 Unless you've encountered an unusual performance problem in a particular
370 version, you'll generally find better features, reliability, and security
371 in a new release and that these benefits are far more significant than any
372 performance you might win or lose.
373
374 Alternatives to Django's template language
375 ------------------------------------------
376
377 For nearly all cases, Django's built-in template language is perfectly
378 adequate. However, if the bottlenecks in your Django project seem to lie in the
379 template system and you have exhausted other opportunities to remedy this, a
380 third-party alternative may be the answer.
381
382 Jinja2_ can offer performance improvements, particularly when it comes to
383 speed.
384
385 Alternative template systems vary in the extent to which they share Django's
386 templating language.
387
388 .. note::
389
390 *If* you experience performance issues in templates, the first thing to do
391 is to understand exactly why. Using an alternative template system may
392 prove faster, but the same gains may also be available without going to
393 that trouble - for example, expensive processing and logic in your
394 templates could be done more efficiently in your views.
395
396 Alternative software implementations
397 ------------------------------------
398
399 It may be worth checking whether Python software you're using has been
400 provided in a different implementation that can execute the same code faster.
401
402 However: most performance problems in well-written Django sites aren't at the
403 Python execution level, but rather in inefficient database querying, caching,
404 and templates. If you're relying on poorly-written Python code, your
405 performance problems are unlikely to be solved by having it execute faster.
406
407 Using an alternative implementation may introduce compatibility, deployment,
408 portability, or maintenance issues. It goes without saying that before adopting
409 a non-standard implementation you should ensure it provides sufficient
410 performance gains for your application to outweigh the potential risks.
411
412 With these caveats in mind, you should be aware of:
413
414 `PyPy <https://www.pypy.org/>`_
415 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
416
417 `PyPy <https://www.pypy.org/>`_ is an implementation of Python in Python itself
418 (the 'standard' Python implementation is in C). PyPy can offer substantial
419 performance gains, typically for heavyweight applications.
420
421 A key aim of the PyPy project is `compatibility
422 <https://www.pypy.org/compat.html>`_ with existing Python APIs and libraries.
423 Django is compatible, but you will need to check the compatibility of other
424 libraries you rely on.
425
426 C implementations of Python libraries
427 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
428
429 Some Python libraries are also implemented in C, and can be much faster. They
430 aim to offer the same APIs. Note that compatibility issues and behavior
431 differences are not unknown (and not always immediately evident).
432
433 .. _Jinja2: https://jinja.palletsprojects.com/