Debian Codegraph @

rfc2277.html

raw
1 <pre>Network Working Group H. Alvestrand
2 Request for Comments: 2277 UNINETT
3 BCP: 18 January 1998
4 Category: Best Current Practice
5
6
7 <span class="h1">IETF Policy on Character Sets and Languages</span>
8
9 Status of this Memo
10
11 This document specifies an Internet Best Current Practices for the
12 Internet Community, and requests discussion and suggestions for
13 improvements. Distribution of this memo is unlimited.
14
15 Copyright Notice
16
17 Copyright (C) The Internet Society (1998). All Rights Reserved.
18
19 <span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>. Introduction</span>
20
21 The Internet is international.
22
23 With the international Internet follows an absolute requirement to
24 interchange data in a multiplicity of languages, which in turn
25 utilize a bewildering number of characters.
26
27 This document is the current policies being applied by the Internet
28 Engineering Steering Group (IESG) towards the standardization efforts
29 in the Internet Engineering Task Force (IETF) in order to help
30 Internet protocols fulfill these requirements.
31
32 The document is very much based upon the recommendations of the IAB
33 Character Set Workshop of February 29-March 1, 1996, which is
34 documented in <a href="./rfc2130">RFC 2130</a> [<a href="#ref-WR" title="&quot;The Report of the IAB Character Set Workshop held 29 February - 1 March, 1996&quot;">WR</a>]. This document attempts to be concise,
35 explicit and clear; people wanting more background are encouraged to
36 read <a href="./rfc2130">RFC 2130</a>.
37
38 The document uses the terms 'MUST', 'SHOULD' and 'MAY', and their
39 negatives, in the way described in [<a href="./rfc2119">RFC 2119</a>]. In this case, 'the
40 specification' as used by <a href="./rfc2119">RFC 2119</a> refers to the processing of
41 protocols being submitted to the IETF standards process.
42
43
44
45
46
47
48
49
50
51
52 <span class="grey">Alvestrand Best Current Practice [Page 1]</span></pre>
53 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span>
54 <span class="grey"><a href="./rfc2277">RFC 2277</a> Charset Policy January 1998</span>
55
56
57 <span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>. Where to do internationalization</span>
58
59 Internationalization is for humans. This means that protocols are not
60 subject to internationalization; text strings are. Where protocol
61 elements look like text tokens, such as in many IETF application
62 layer protocols, protocols MUST specify which parts are protocol and
63 which are text. [WR 2.2.1.1]
64
65 Names are a problem, because people feel strongly about them, many of
66 them are mostly for local usage, and all of them tend to leak out of
67 the local context at times. <a href="./rfc1958">RFC 1958</a> [<a href="./rfc1958">RFC 1958</a>] recommends US-ASCII
68 for all globally visible names.
69
70 This document does not mandate a policy on name internationalization,
71 but requires that all protocols describe whether names are
72 internationalized or US-ASCII.
73
74 NOTE: In the protocol stack for any given application, there is
75 usually one or a few layers that need to address these problems.
76
77 It would, for instance, not be appropriate to define language tags
78 for Ethernet frames. But it is the responsibility of the WGs to
79 ensure that whenever responsibility for internationalization is left
80 to "another layer", those responsible for that layer are in fact
81 aware that they HAVE that responsibility.
82
83 <span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>. Definition of Terms</span>
84
85 This document uses the term "charset" to mean a set of rules for
86 mapping from a sequence of octets to a sequence of characters, such
87 as the combination of a coded character set and a character encoding
88 scheme; this is also what is used as an identifier in MIME "charset="
89 parameters, and registered in the IANA charset registry [<a href="#ref-REG">REG</a>]. (Note
90 that this is NOT a term used by other standards bodies, such as ISO).
91
92 For a definition of the term "coded character set", refer to the
93 workshop report.
94
95 A "name" is an identifier such as a person's name, a hostname, a
96 domainname, a filename or an E-mail address; it is often treated as
97 an identifier rather than as a piece of text, and is often used in
98 protocols as an identifier for entities, without surrounding text.
99
100 <span class="h3"><a class="selflink" id="section-3.1" href="#section-3.1">3.1</a>. What charset to use</span>
101
102 All protocols MUST identify, for all character data, which charset is
103 in use.
104
105
106
107
108 <span class="grey">Alvestrand Best Current Practice [Page 2]</span></pre>
109 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span>
110 <span class="grey"><a href="./rfc2277">RFC 2277</a> Charset Policy January 1998</span>
111
112
113 Protocols MUST be able to use the UTF-8 charset, which consists of
114 the ISO 10646 coded character set combined with the UTF-8 character
115 encoding scheme, as defined in [<a href="#ref-10646">10646</a>] Annex R (published in
116 Amendment 2), for all text.
117
118 Protocols MAY specify, in addition, how to use other charsets or
119 other character encoding schemes for ISO 10646, such as UTF-16, but
120 lack of an ability to use UTF-8 is a violation of this policy; such a
121 violation would need a variance procedure (<a href="https://www.rfc-editor.org/bcp/bcp9#section-9">[BCP9] section&nbsp;9</a>) with
122 clear and solid justification in the protocol specification document
123 before being entered into or advanced upon the standards track.
124
125 For existing protocols or protocols that move data from existing
126 datastores, support of other charsets, or even using a default other
127 than UTF-8, may be a requirement. This is acceptable, but UTF-8
128 support MUST be possible.
129
130 When using other charsets than UTF-8, these MUST be registered in the
131 IANA charset registry, if necessary by registering them when the
132 protocol is published.
133
134 (Note: ISO 10646 calls the UTF-8 CES a "Transformation Format" rather
135 than a "character encoding scheme", but it fits the charset workshop
136 report definition of a character encoding scheme).
137
138 <span class="h3"><a class="selflink" id="section-3.2" href="#section-3.2">3.2</a>. How to decide a charset</span>
139
140 When the protocol allows a choice of multiple charsets, someone must
141 make a decision on which charset to use.
142
143 In some cases, like HTTP, there is direct or semi-direct
144 communication between the producer and the consumer of data
145 containing text. In such cases, it may make sense to negotiate a
146 charset before sending data.
147
148 In other cases, like E-mail or stored data, there is no such
149 communication, and the best one can do is to make sure the charset is
150 clearly identified with the stored data, and choosing a charset that
151 is as widely known as possible.
152
153 Note that a charset is an absolute; text that is encoded in a charset
154 cannot be rendered comprehensibly without supporting that charset.
155
156 (This also applies to English texts; charsets like EBCDIC do NOT have
157 ASCII as a proper subset)
158
159
160
161
162
163
164 <span class="grey">Alvestrand Best Current Practice [Page 3]</span></pre>
165 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span>
166 <span class="grey"><a href="./rfc2277">RFC 2277</a> Charset Policy January 1998</span>
167
168
169 Negotiating a charset may be regarded as an interim mechanism that is
170 to be supported until support for interchange of UTF-8 is prevalent;
171 however, the timeframe of "interim" may be at least 50 years, so
172 there is every reason to think of it as permanent in practice.
173
174 <span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>. Languages</span>
175
176 <span class="h3"><a class="selflink" id="section-4.1" href="#section-4.1">4.1</a>. The need for language information</span>
177
178 All human-readable text has a language.
179
180 Many operations, including high quality formatting, text-to-speech
181 synthesis, searching, hyphenation, spellchecking and so on benefit
182 greatly from access to information about the language of a piece of
183 text. [WC 3.1.1.4].
184
185 Humans have some tolerance for foreign languages, but are generally
186 very unhappy with being presented text in a language they do not
187 understand; this is why negotiation of language is needed.
188
189 In most cases, machines will not be able to deduce the language of a
190 transmitted text by themselves; the protocol must specify how to
191 transfer the language information if it is to be available at all.
192
193 The interaction between language and processing is complex; for
194 instance, if I compare "name-of-thing(lang=en)" to "name-of-
195 thing(lang=no)" for equality, I will generally expect a match, while
196 the word "ask(no)" is a kind of tree, and is hardly useful as a
197 command verb.
198
199 <span class="h3"><a class="selflink" id="section-4.2" href="#section-4.2">4.2</a>. Requirement for language tagging</span>
200
201 Protocols that transfer text MUST provide for carrying information
202 about the language of that text.
203
204 Protocols SHOULD also provide for carrying information about the
205 language of names, where appropriate.
206
207 Note that this does NOT mean that such information must always be
208 present; the requirement is that if the sender of information wishes
209 to send information about the language of a text, the protocol
210 provides a well-defined way to carry this information.
211
212
213
214
215
216
217
218
219
220 <span class="grey">Alvestrand Best Current Practice [Page 4]</span></pre>
221 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span>
222 <span class="grey"><a href="./rfc2277">RFC 2277</a> Charset Policy January 1998</span>
223
224
225 <span class="h3"><a class="selflink" id="section-4.3" href="#section-4.3">4.3</a>. How to identify a language</span>
226
227 The <a href="./rfc1766">RFC 1766</a> language tag is at the moment the most flexible tool
228 available for identifying a language; protocols SHOULD use this, or
229 provide clear and solid justification for doing otherwise in the
230 document.
231
232 Note also that a language is distinct from a POSIX locale; a POSIX
233 locale identifies a set of cultural conventions, which may imply a
234 language (the POSIX or "C" locale of course do not), while a language
235 tag as described in <a href="./rfc1766">RFC 1766</a> identifies only a language.
236
237 <span class="h3"><a class="selflink" id="section-4.4" href="#section-4.4">4.4</a>. Considerations for language negotiation</span>
238
239 Protocols where users have text presented to them in response to user
240 actions MUST provide for support of multiple languages.
241
242 How this is done will vary between protocols; for instance, in some
243 cases, a negotiation where the client proposes a set of languages and
244 the server replies with one is appropriate; in other cases, a server
245 may choose to send multiple variants of a text and let the client
246 pick which one to display.
247
248 Negotiation is useful in the case where one side of the protocol
249 exchange is able to present text in multiple languages to the other
250 side, and the other side has a preference for one of these; the most
251 common example is the text part of error responses, or Web pages that
252 are available in multiple languages.
253
254 Negotiating a language should be regarded as a permanent requirement
255 of the protocol that will not go away at any time in the future.
256
257 In many cases, it should be possible to include it as part of the
258 connection establishment, together with authentication and other
259 preferences negotiation.
260
261 <span class="h3"><a class="selflink" id="section-4.5" href="#section-4.5">4.5</a>. Default Language</span>
262
263 When human-readable text must be presented in a context where the
264 sender has no knowledge of the recipient's language preferences (such
265 as login failures or E-mailed warnings, or prior to language
266 negotiation), text SHOULD be presented in Default Language.
267
268 Default Language is assigned the tag "i-default" according to the
269 procedures of <a href="./rfc1766">RFC 1766</a>. It is not a specific language, but rather
270 identifies the condition where the language preferences of the user
271 cannot be established.
272
273
274
275
276 <span class="grey">Alvestrand Best Current Practice [Page 5]</span></pre>
277 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span>
278 <span class="grey"><a href="./rfc2277">RFC 2277</a> Charset Policy January 1998</span>
279
280
281 Messages in Default Language MUST be understandable by an English-
282 speaking person, since English is the language which, worldwide, the
283 greatest number of people will be able to get adequate help in
284 interpreting when working with computers.
285
286 Note that negotiating English is NOT the same as Default Language;
287 Default Language is an emergency measure in otherwise unmanageable
288 situations.
289
290 In many cases, using only English text is reasonable; in some cases,
291 the English text may be augumented by text in other languages.
292
293 <span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>. Locale</span>
294
295 The POSIX standard [<a href="#ref-POSIX">POSIX</a>] defines a concept called a "locale", which
296 includes a lot of information about collating order for sorting, date
297 format, currency format and so on.
298
299 In some cases, and especially with text where the user is expected to
300 do processing on the text, locale information may be usefully
301 attached to the text; this would identify the sender's opinion about
302 appropriate rules to follow when processing the document, which the
303 recipient may choose to agree with or ignore.
304
305 This document does not require the communication of locale
306 information on all text, but encourages its inclusion when
307 appropriate.
308
309 Note that language and character set information will often be
310 present as parts of a locale tag (such as no_NO.iso-8859-1; the
311 language is before the underscore and the character set is after the
312 dot); care must be taken to define precisely which specification of
313 character set and language applies to any one text item.
314
315 The default locale is the "POSIX" locale.
316
317 <span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>. Documenting internationalization decisions</span>
318
319 In documents that deal with internationalization issues at all, a
320 synopsis of the approaches chosen for internationalization SHOULD be
321 collected into a section called "Internationalization
322 considerations", and placed next to the Security Considerations
323 section.
324
325 This provides an easy reference for those who are looking for advice
326 on these issues when implementing the protocol.
327
328
329
330
331
332 <span class="grey">Alvestrand Best Current Practice [Page 6]</span></pre>
333 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span>
334 <span class="grey"><a href="./rfc2277">RFC 2277</a> Charset Policy January 1998</span>
335
336
337 <span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>. Security Considerations</span>
338
339 Apart from the fact that security warnings in a foreign language may
340 cause inappropriate behaviour from the user, and the fact that
341 multilingual systems usually have problems with consistency between
342 language variants, no security considerations relevant have been
343 identified.
344
345 <span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>. References</span>
346
347 [<a id="ref-10646">10646</a>]
348 ISO/IEC, Information Technology - Universal Multiple-Octet Coded
349 Character Set (UCS) - Part 1: Architecture and Basic
350 Multilingual Plane, May 1993, with amendments
351
352 [<a id="ref-RFC 2119">RFC 2119</a>]
353 Bradner, S., "Key words for use in RFCs to Indicate Requirement
354 Levels", <a href="https://www.rfc-editor.org/bcp/bcp14">BCP 14</a>, <a href="./rfc2119">RFC 2119</a>, March 1997.
355
356 [<a id="ref-WR">WR</a>] Weider, C., Preston, C., Simonsen, K., Alvestrand, H,
357 Atkinson, R., Crispin, M., and P. Svanberg, "The Report of the
358 IAB Character Set Workshop held 29 February - 1 March, 1996",
359 <a href="./rfc2130">RFC 2130</a>, April 1997.
360
361 [<a id="ref-RFC 1958">RFC 1958</a>]
362 Carpenter, B., "Architectural Principles of the Internet", <a href="./rfc1958">RFC</a>
363 <a href="./rfc1958">1958</a>, June 1996.
364
365 [<a id="ref-POSIX">POSIX</a>]
366 ISO/IEC 9945-2:1993 Information technology -- Portable Operating
367 System Interface (POSIX) -- Part 2: Shell and Utilities
368
369 [<a id="ref-REG">REG</a>]
370 Freed, N., and J. Postel, "IANA Charset Registration
371 Procedures", <a href="https://www.rfc-editor.org/bcp/bcp19">BCP 19</a>, <a href="./rfc2278">RFC 2278</a>, January 1998.
372
373 [<a id="ref-UTF-8">UTF-8</a>]
374 Yergeau, F., "UTF-8, a transformation format of ISO 10646", <a href="./rfc2279">RFC</a>
375 <a href="./rfc2279">2279</a>, January 1998.
376
377 [<a id="ref-BCP9">BCP9</a>]
378 Bradner, S., "The Internet Standards Process -- Revision 3," <a href="https://www.rfc-editor.org/bcp/bcp9">BCP</a>
379 <a href="https://www.rfc-editor.org/bcp/bcp9">9</a>, <a href="./rfc2026">RFC 2026</a>, October 1996.
380
381
382
383
384
385
386
387
388 <span class="grey">Alvestrand Best Current Practice [Page 7]</span></pre>
389 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-8" ></span>
390 <span class="grey"><a href="./rfc2277">RFC 2277</a> Charset Policy January 1998</span>
391
392
393 <span class="h2"><a class="selflink" id="section-9" href="#section-9">9</a>. Author's Address</span>
394
395 Harald Tveit Alvestrand
396 UNINETT
397 P.O.Box 6883 Elgeseter
398 N-7002 TRONDHEIM
399 NORWAY
400
401 Phone: +47 73 59 70 94
402 EMail: [email protected]
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444 <span class="grey">Alvestrand Best Current Practice [Page 8]</span></pre>
445 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-9" ></span>
446 <span class="grey"><a href="./rfc2277">RFC 2277</a> Charset Policy January 1998</span>
447
448
449 <span class="h2"><a class="selflink" id="section-10" href="#section-10">10</a>. Full Copyright Statement</span>
450
451 Copyright (C) The Internet Society (1998). All Rights Reserved.
452
453 This document and translations of it may be copied and furnished to
454 others, and derivative works that comment on or otherwise explain it
455 or assist in its implementation may be prepared, copied, published
456 and distributed, in whole or in part, without restriction of any
457 kind, provided that the above copyright notice and this paragraph are
458 included on all such copies and derivative works. However, this
459 document itself may not be modified in any way, such as by removing
460 the copyright notice or references to the Internet Society or other
461 Internet organizations, except as needed for the purpose of
462 developing Internet standards in which case the procedures for
463 copyrights defined in the Internet Standards process must be
464 followed, or as required to translate it into languages other than
465 English.
466
467 The limited permissions granted above are perpetual and will not be
468 revoked by the Internet Society or its successors or assigns.
469
470 This document and the information contained herein is provided on an
471 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
472 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
473 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
474 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
475 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500 Alvestrand Best Current Practice [Page 9]
501 </pre>