Debian Codegraph @

docs/ref/unicode.txt

raw
1 ============
2 Unicode data
3 ============
4
5 Django supports Unicode data everywhere.
6
7 This document tells you what you need to know if you're writing applications
8 that use data or templates that are encoded in something other than ASCII.
9
10 Creating the database
11 =====================
12
13 Make sure your database is configured to be able to store arbitrary string
14 data. Normally, this means giving it an encoding of UTF-8 or UTF-16. If you use
15 a more restrictive encoding -- for example, latin1 (iso8859-1) -- you won't be
16 able to store certain characters in the database, and information will be lost.
17
18 * MySQL users, refer to the `MySQL manual`_ for details on how to set or alter
19 the database character set encoding.
20
21 * PostgreSQL users, refer to the `PostgreSQL manual`_ for details on creating
22 databases with the correct encoding.
23
24 * Oracle users, refer to the `Oracle manual`_ for details on how to set
25 (`section 2`_) or alter (`section 11`_) the database character set encoding.
26
27 * SQLite users, there is nothing you need to do. SQLite always uses UTF-8
28 for internal encoding.
29
30 .. _MySQL manual: https://dev.mysql.com/doc/refman/en/charset-database.html
31 .. _PostgreSQL manual: https://www.postgresql.org/docs/current/multibyte.html#MULTIBYTE-SETTING
32 .. _Oracle manual: https://docs.oracle.com/en/database/oracle/oracle-database/21/nlspg/index.html
33 .. _section 2: https://docs.oracle.com/en/database/oracle/oracle-database/21/nlspg/choosing-character-set.html
34 .. _section 11: https://docs.oracle.com/en/database/oracle/oracle-database/21/nlspg/character-set-migration.html
35
36 All of Django's database backends automatically convert strings into
37 the appropriate encoding for talking to the database. They also automatically
38 convert strings retrieved from the database into strings. You don't even need
39 to tell Django what encoding your database uses: that is handled transparently.
40
41 For more, see the section "The database API" below.
42
43 General string handling
44 =======================
45
46 Whenever you use strings with Django -- e.g., in database lookups, template
47 rendering or anywhere else -- you have two choices for encoding those strings.
48 You can use normal strings or bytestrings (starting with a 'b').
49
50 .. warning::
51
52 A bytestring does not carry any information with it about its encoding.
53 For that reason, we have to make an assumption, and Django assumes that all
54 bytestrings are in UTF-8.
55
56 If you pass a string to Django that has been encoded in some other format,
57 things will go wrong in interesting ways. Usually, Django will raise a
58 ``UnicodeDecodeError`` at some point.
59
60 If your code only uses ASCII data, it's safe to use your normal strings,
61 passing them around at will, because ASCII is a subset of UTF-8.
62
63 Don't be fooled into thinking that if your :setting:`DEFAULT_CHARSET` setting is set
64 to something other than ``'utf-8'`` you can use that other encoding in your
65 bytestrings! :setting:`DEFAULT_CHARSET` only applies to the strings generated as
66 the result of template rendering (and email). Django will always assume UTF-8
67 encoding for internal bytestrings. The reason for this is that the
68 :setting:`DEFAULT_CHARSET` setting is not actually under your control (if you are the
69 application developer). It's under the control of the person installing and
70 using your application -- and if that person chooses a different setting, your
71 code must still continue to work. Ergo, it cannot rely on that setting.
72
73 In most cases when Django is dealing with strings, it will convert them to
74 strings before doing anything else. So, as a general rule, if you pass
75 in a bytestring, be prepared to receive a string back in the result.
76
77 Translated strings
78 ------------------
79
80 Aside from strings and bytestrings, there's a third type of string-like
81 object you may encounter when using Django. The framework's
82 internationalization features introduce the concept of a "lazy translation" --
83 a string that has been marked as translated but whose actual translation result
84 isn't determined until the object is used in a string. This feature is useful
85 in cases where the translation locale is unknown until the string is used, even
86 though the string might have originally been created when the code was first
87 imported.
88
89 Normally, you won't have to worry about lazy translations. Just be aware that
90 if you examine an object and it claims to be a
91 ``django.utils.functional.__proxy__`` object, it is a lazy translation.
92 Calling ``str()`` with the lazy translation as the argument will generate a
93 string in the current locale.
94
95 For more details about lazy translation objects, refer to the
96 :doc:`internationalization </topics/i18n/index>` documentation.
97
98 Useful utility functions
99 ------------------------
100
101 Because some string operations come up again and again, Django ships with a few
102 useful functions that should make working with string and bytestring objects
103 a bit easier.
104
105 Conversion functions
106 ~~~~~~~~~~~~~~~~~~~~
107
108 The ``django.utils.encoding`` module contains a few functions that are handy
109 for converting back and forth between strings and bytestrings.
110
111 * ``smart_str(s, encoding='utf-8', strings_only=False, errors='strict')``
112 converts its input to a string. The ``encoding`` parameter
113 specifies the input encoding. (For example, Django uses this internally
114 when processing form input data, which might not be UTF-8 encoded.) The
115 ``strings_only`` parameter, if set to True, will result in Python
116 numbers, booleans and ``None`` not being converted to a string (they keep
117 their original types). The ``errors`` parameter takes any of the values
118 that are accepted by Python's ``str()`` function for its error
119 handling.
120
121 * ``force_str(s, encoding='utf-8', strings_only=False, errors='strict')`` is
122 identical to ``smart_str()`` in almost all cases. The difference is when the
123 first argument is a :ref:`lazy translation <lazy-translations>` instance.
124 While ``smart_str()`` preserves lazy translations, ``force_str()`` forces
125 those objects to a string (causing the translation to occur). Normally,
126 you'll want to use ``smart_str()``. However, ``force_str()`` is useful in
127 template tags and filters that absolutely *must* have a string to work with,
128 not just something that can be converted to a string.
129
130 * ``smart_bytes(s, encoding='utf-8', strings_only=False, errors='strict')``
131 is essentially the opposite of ``smart_str()``. It forces the first
132 argument to a bytestring. The ``strings_only`` parameter has the same
133 behavior as for ``smart_str()`` and ``force_str()``. This is
134 slightly different semantics from Python's builtin ``str()`` function,
135 but the difference is needed in a few places within Django's internals.
136
137 Normally, you'll only need to use ``force_str()``. Call it as early as
138 possible on any input data that might be either a string or a bytestring, and
139 from then on, you can treat the result as always being a string.
140
141 .. _uri-and-iri-handling:
142
143 URI and IRI handling
144 ~~~~~~~~~~~~~~~~~~~~
145
146 Web frameworks have to deal with URLs (which are a type of IRI). One
147 requirement of URLs is that they are encoded using only ASCII characters.
148 However, in an international environment, you might need to construct a
149 URL from an :rfc:`IRI <3987>` -- very loosely speaking, a :rfc:`URI <3986>`
150 that can contain Unicode characters. Use these functions for quoting and
151 converting an IRI to a URI:
152
153 * The :func:`django.utils.encoding.iri_to_uri()` function, which implements the
154 conversion from IRI to URI as required by :rfc:`3987#section-3.1`.
155
156 * The :func:`urllib.parse.quote` and :func:`urllib.parse.quote_plus`
157 functions from Python's standard library.
158
159 These two groups of functions have slightly different purposes, and it's
160 important to keep them straight. Normally, you would use ``quote()`` on the
161 individual portions of the IRI or URI path so that any reserved characters
162 such as '&' or '%' are correctly encoded. Then, you apply ``iri_to_uri()`` to
163 the full IRI and it converts any non-ASCII characters to the correct encoded
164 values.
165
166 .. note::
167 Technically, it isn't correct to say that ``iri_to_uri()`` implements the
168 full algorithm in the IRI specification. It doesn't (yet) perform the
169 international domain name encoding portion of the algorithm.
170
171 The ``iri_to_uri()`` function will not change ASCII characters that are
172 otherwise permitted in a URL. So, for example, the character '%' is not
173 further encoded when passed to ``iri_to_uri()``. This means you can pass a
174 full URL to this function and it will not mess up the query string or anything
175 like that.
176
177 An example might clarify things here:
178
179 .. code-block:: pycon
180
181 >>> from urllib.parse import quote
182 >>> from django.utils.encoding import iri_to_uri
183 >>> quote("Paris & Orléans")
184 'Paris%20%26%20Orl%C3%A9ans'
185 >>> iri_to_uri("/favorites/François/%s" % quote("Paris & Orléans"))
186 '/favorites/Fran%C3%A7ois/Paris%20%26%20Orl%C3%A9ans'
187
188 If you look carefully, you can see that the portion that was generated by
189 ``quote()`` in the second example was not double-quoted when passed to
190 ``iri_to_uri()``. This is a very important and useful feature. It means that
191 you can construct your IRI without worrying about whether it contains
192 non-ASCII characters and then, right at the end, call ``iri_to_uri()`` on the
193 result.
194
195 Similarly, Django provides :func:`django.utils.encoding.uri_to_iri()` which
196 implements the conversion from URI to IRI as per :rfc:`3987#section-3.2`.
197
198 An example to demonstrate:
199
200 .. code-block:: pycon
201
202 >>> from django.utils.encoding import uri_to_iri
203 >>> uri_to_iri("/%E2%99%A5%E2%99%A5/?utf8=%E2%9C%93")
204 '/♥♥/?utf8=✓'
205 >>> uri_to_iri("%A9hello%3Fworld")
206 '%A9hello%3Fworld'
207
208 In the first example, the UTF-8 characters are unquoted. In the second, the
209 percent-encodings remain unchanged because they lie outside the valid UTF-8
210 range or represent a reserved character.
211
212 Both ``iri_to_uri()`` and ``uri_to_iri()`` functions are idempotent, which means the
213 following is always true::
214
215 iri_to_uri(iri_to_uri(some_string)) == iri_to_uri(some_string)
216 uri_to_iri(uri_to_iri(some_string)) == uri_to_iri(some_string)
217
218 So you can safely call it multiple times on the same URI/IRI without risking
219 double-quoting problems.
220
221 Models
222 ======
223
224 Because all strings are returned from the database as ``str`` objects, model
225 fields that are character based (CharField, TextField, URLField, etc.) will
226 contain Unicode values when Django retrieves data from the database. This
227 is *always* the case, even if the data could fit into an ASCII bytestring.
228
229 You can pass in bytestrings when creating a model or populating a field, and
230 Django will convert it to strings when it needs to.
231
232 Taking care in ``get_absolute_url()``
233 -------------------------------------
234
235 URLs can only contain ASCII characters. If you're constructing a URL from
236 pieces of data that might be non-ASCII, be careful to encode the results in a
237 way that is suitable for a URL. The :func:`~django.urls.reverse` function
238 handles this for you automatically.
239
240 If you're constructing a URL manually (i.e., *not* using the ``reverse()``
241 function), you'll need to take care of the encoding yourself. In this case,
242 use the ``iri_to_uri()`` and ``quote()`` functions that were documented
243 above_. For example::
244
245 from urllib.parse import quote
246 from django.utils.encoding import iri_to_uri
247
248
249 def get_absolute_url(self):
250 url = "/person/%s/?x=0&y=0" % quote(self.location)
251 return iri_to_uri(url)
252
253 This function returns a correctly encoded URL even if ``self.location`` is
254 something like "Jack visited Paris & Orléans". (In fact, the ``iri_to_uri()``
255 call isn't strictly necessary in the above example, because all the
256 non-ASCII characters would have been removed in quoting in the first line.)
257
258 .. _above: `URI and IRI handling`_
259
260 Templates
261 =========
262
263 Use strings when creating templates manually::
264
265 from django.template import Template
266
267 t2 = Template("This is a string template.")
268
269 But the common case is to read templates from the filesystem. If your template
270 files are not stored with a UTF-8 encoding, adjust the :setting:`TEMPLATES`
271 setting. The built-in :py:mod:`~django.template.backends.django` backend
272 provides the ``'file_charset'`` option to change the encoding used to read
273 files from disk.
274
275 The :setting:`DEFAULT_CHARSET` setting controls the encoding of rendered templates.
276 This is set to UTF-8 by default.
277
278 Template tags and filters
279 -------------------------
280
281 A couple of tips to remember when writing your own template tags and filters:
282
283 * Always return strings from a template tag's ``render()`` method
284 and from template filters.
285
286 * Use ``force_str()`` in preference to ``smart_str()`` in these
287 places. Tag rendering and filter calls occur as the template is being
288 rendered, so there is no advantage to postponing the conversion of lazy
289 translation objects into strings. It's easier to work solely with
290 strings at that point.
291
292 .. _unicode-files:
293
294 Files
295 =====
296
297 If you intend to allow users to upload files, you must ensure that the
298 environment used to run Django is configured to work with non-ASCII file names.
299 If your environment isn't configured correctly, you'll encounter
300 ``UnicodeEncodeError`` exceptions when saving files with file names or content
301 that contains non-ASCII characters.
302
303 Filesystem support for UTF-8 file names varies and might depend on the
304 environment. Check your current configuration in an interactive Python shell by
305 running::
306
307 import sys
308
309 sys.getfilesystemencoding()
310
311 This should output "UTF-8".
312
313 The ``LANG`` environment variable is responsible for setting the expected
314 encoding on Unix platforms. Consult the documentation for your operating system
315 and application server for the appropriate syntax and location to set this
316 variable. See the :doc:`/howto/deployment/wsgi/modwsgi` for examples.
317
318 In your development environment, you might need to add a setting to your
319 ``~.bashrc`` analogous to:
320
321 .. code-block:: shell
322
323 export LANG="en_US.UTF-8"
324
325 Form submission
326 ===============
327
328 HTML form submission is a tricky area. There's no guarantee that the
329 submission will include encoding information, which means the framework might
330 have to guess at the encoding of submitted data.
331
332 Django adopts a "lazy" approach to decoding form data. The data in an
333 ``HttpRequest`` object is only decoded when you access it. In fact, most of
334 the data is not decoded at all. Only the ``HttpRequest.GET`` and
335 ``HttpRequest.POST`` data structures have any decoding applied to them. Those
336 two fields will return their members as Unicode data. All other attributes and
337 methods of ``HttpRequest`` return data exactly as it was submitted by the
338 client.
339
340 By default, the :setting:`DEFAULT_CHARSET` setting is used as the assumed encoding
341 for form data. If you need to change this for a particular form, you can set
342 the ``encoding`` attribute on an ``HttpRequest`` instance. For example::
343
344 def some_view(request):
345 # We know that the data must be encoded as KOI8-R (for some reason).
346 request.encoding = "koi8-r"
347 ...
348
349 You can even change the encoding after having accessed ``request.GET`` or
350 ``request.POST``, and all subsequent accesses will use the new encoding.
351
352 Most developers won't need to worry about changing form encoding, but this is
353 a useful feature for applications that talk to legacy systems whose encoding
354 you cannot control.
355
356 Django does not decode the data of file uploads, because that data is normally
357 treated as collections of bytes, rather than strings. Any automatic decoding
358 there would alter the meaning of the stream of bytes.