Debian Codegraph @

upstream/mageia-cauldron/man3pm/Locale::Maketext::TPJ13.3pm

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "Locale::Maketext::TPJ13 3pm"
61 .TH Locale::Maketext::TPJ13 3pm 2023-12-22 "perl v5.42.0" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 Locale::Maketext::TPJ13 \-\- article about software localization
68 .SH SYNOPSIS
69 .IX Header "SYNOPSIS"
70 .Vb 1
71 \& # This an article, not a module.
72 .Ve
73 .SH DESCRIPTION
74 .IX Header "DESCRIPTION"
75 The following article by Sean M. Burke and Jordan Lachler
76 first appeared in \fIThe Perl Journal\fR #13
77 and is copyright 1999 The Perl Journal. It appears
78 courtesy of Jon Orwant and The Perl Journal. This document may be
79 distributed under the same terms as Perl itself.
80 .SH "Localization and Perl: gettext breaks, Maketext fixes"
81 .IX Header "Localization and Perl: gettext breaks, Maketext fixes"
82 by Sean M. Burke and Jordan Lachler
83 .PP
84 This article points out cases where gettext (a common system for
85 localizing software interfaces \-\- i.e., making them work in the user\*(Aqs
86 language of choice) fails because of basic differences between human
87 languages. This article then describes Maketext, a new system capable
88 of correctly treating these differences.
89 .SS "A Localization Horror Story: It Could Happen To You"
90 .IX Subsection "A Localization Horror Story: It Could Happen To You"
91 .RS 4
92 "There are a number of languages spoken by human beings in this
93 world."
94 .Sp
95 \&\-\- Harald Tveit Alvestrand, in RFC 1766, "Tags for the
96 Identification of Languages"
97 .RE
98 .PP
99 Imagine that your task for the day is to localize a piece of software
100 \&\-\- and luckily for you, the only output the program emits is two
101 messages, like this:
102 .PP
103 .Vb 1
104 \& I scanned 12 directories.
105 \&
106 \& Your query matched 10 files in 4 directories.
107 .Ve
108 .PP
109 So how hard could that be? You look at the code that
110 produces the first item, and it reads:
111 .PP
112 .Vb 2
113 \& printf("I scanned %g directories.",
114 \& $directory_count);
115 .Ve
116 .PP
117 You think about that, and realize that it doesn\*(Aqt even work right for
118 English, as it can produce this output:
119 .PP
120 .Vb 1
121 \& I scanned 1 directories.
122 .Ve
123 .PP
124 So you rewrite it to read:
125 .PP
126 .Vb 5
127 \& printf("I scanned %g %s.",
128 \& $directory_count,
129 \& $directory_count == 1 ?
130 \& "directory" : "directories",
131 \& );
132 .Ve
133 .PP
134 \&...which does the Right Thing. (In case you don\*(Aqt recall, "%g" is for
135 locale\-specific number interpolation, and "%s" is for string
136 interpolation.)
137 .PP
138 But you still have to localize it for all the languages you\*(Aqre
139 producing this software for, so you pull Locale::gettext off of CPAN
140 so you can access the \f(CW\*(C`gettext\*(C'\fR C functions you\*(Aqve heard are standard
141 for localization tasks.
142 .PP
143 And you write:
144 .PP
145 .Vb 5
146 \& printf(gettext("I scanned %g %s."),
147 \& $dir_scan_count,
148 \& $dir_scan_count == 1 ?
149 \& gettext("directory") : gettext("directories"),
150 \& );
151 .Ve
152 .PP
153 But you then read in the gettext manual (Drepper, Miller, and Pinard 1995)
154 that this is not a good idea, since how a single word like "directory"
155 or "directories" is translated may depend on context \-\- and this is
156 true, since in a case language like German or Russian, you\*(Aqd may need
157 these words with a different case ending in the first instance (where the
158 word is the object of a verb) than in the second instance, which you haven\*(Aqt even
159 gotten to yet (where the word is the object of a preposition, "in \f(CW%g\fR
160 directories") \-\- assuming these keep the same syntax when translated
161 into those languages.
162 .PP
163 So, on the advice of the gettext manual, you rewrite:
164 .PP
165 .Vb 4
166 \& printf( $dir_scan_count == 1 ?
167 \& gettext("I scanned %g directory.") :
168 \& gettext("I scanned %g directories."),
169 \& $dir_scan_count );
170 .Ve
171 .PP
172 So, you email your various translators (the boss decides that the
173 languages du jour are Chinese, Arabic, Russian, and Italian, so you
174 have one translator for each), asking for translations for "I scanned
175 \&\f(CW%g\fR directory." and "I scanned \f(CW%g\fR directories.". When they reply,
176 you\*(Aqll put that in the lexicons for gettext to use when it localizes
177 your software, so that when the user is running under the "zh"
178 (Chinese) locale, gettext("I scanned \f(CW%g\fR directory.") will return the
179 appropriate Chinese text, with a "%g" in there where printf can then
180 interpolate \f(CW$dir_scan\fR.
181 .PP
182 Your Chinese translator emails right back \-\- he says both of these
183 phrases translate to the same thing in Chinese, because, in linguistic
184 jargon, Chinese "doesn\*(Aqt have number as a grammatical category" \-\-
185 whereas English does. That is, English has grammatical rules that
186 refer to "number", i.e., whether something is grammatically singular
187 or plural; and one of these rules is the one that forces nouns to take
188 a plural suffix (generally "s") when in a plural context, as they are when
189 they follow a number other than "one" (including, oddly enough, "zero").
190 Chinese has no such rules, and so has just the one phrase where English
191 has two. But, no problem, you can have this one Chinese phrase appear
192 as the translation for the two English phrases in the "zh" gettext
193 lexicon for your program.
194 .PP
195 Emboldened by this, you dive into the second phrase that your software
196 needs to output: "Your query matched 10 files in 4 directories.". You notice
197 that if you want to treat phrases as indivisible, as the gettext
198 manual wisely advises, you need four cases now, instead of two, to
199 cover the permutations of singular and plural on the two items,
200 \&\f(CW$dir_count\fR and \f(CW$file_count\fR. So you try this:
201 .PP
202 .Vb 9
203 \& printf( $file_count == 1 ?
204 \& ( $directory_count == 1 ?
205 \& gettext("Your query matched %g file in %g directory.") :
206 \& gettext("Your query matched %g file in %g directories.") ) :
207 \& ( $directory_count == 1 ?
208 \& gettext("Your query matched %g files in %g directory.") :
209 \& gettext("Your query matched %g files in %g directories.") ),
210 \& $file_count, $directory_count,
211 \& );
212 .Ve
213 .PP
214 (The case of "1 file in 2 [or more] directories" could, I suppose,
215 occur in the case of symlinking or something of the sort.)
216 .PP
217 It occurs to you that this is not the prettiest code you\*(Aqve ever
218 written, but this seems the way to go. You mail off to the
219 translators asking for translations for these four cases. The
220 Chinese guy replies with the one phrase that these all translate to in
221 Chinese, and that phrase has two "%g"s in it, as it should \-\- but
222 there\*(Aqs a problem. He translates it word\-for\-word back: "In \f(CW%g\fR
223 directories contains \f(CW%g\fR files match your query." The \f(CW%g\fR
224 slots are in an order reverse to what they are in English. You wonder
225 how you\*(Aqll get gettext to handle that.
226 .PP
227 But you put it aside for the moment, and optimistically hope that the
228 other translators won\*(Aqt have this problem, and that their languages
229 will be better behaved \-\- i.e., that they will be just like English.
230 .PP
231 But the Arabic translator is the next to write back. First off, your
232 code for "I scanned \f(CW%g\fR directory." or "I scanned \f(CW%g\fR directories."
233 assumes there\*(Aqs only singular or plural. But, to use linguistic
234 jargon again, Arabic has grammatical number, like English (but unlike
235 Chinese), but it\*(Aqs a three\-term category: singular, dual, and plural.
236 In other words, the way you say "directory" depends on whether there\*(Aqs
237 one directory, or \fItwo\fR of them, or \fImore than two\fR of them. Your
238 test of \f(CW\*(C`($directory == 1)\*(C'\fR no longer does the job. And it means
239 that where English\*(Aqs grammatical category of number necessitates
240 only the two permutations of the first sentence based on "directory
241 [singular]" and "directories [plural]", Arabic has three \-\- and,
242 worse, in the second sentence ("Your query matched \f(CW%g\fR file in \f(CW%g\fR
243 directory."), where English has four, Arabic has nine. You sense
244 an unwelcome, exponential trend taking shape.
245 .PP
246 Your Italian translator emails you back and says that "I searched 0
247 directories" (a possible English output of your program) is stilted,
248 and if you think that\*(Aqs fine English, that\*(Aqs your problem, but that
249 \&\fIjust will not do\fR in the language of Dante. He insists that where
250 \&\f(CW$directory_count\fR is 0, your program should produce the Italian text
251 for "I \fIdidn\*(Aqt\fR scan \fIany\fR directories.". And ditto for "I didn\*(Aqt
252 match any files in any directories", although he says the last part
253 about "in any directories" should probably just be left off.
254 .PP
255 You wonder how you\*(Aqll get gettext to handle this; to accommodate the
256 ways Arabic, Chinese, and Italian deal with numbers in just these few
257 very simple phrases, you need to write code that will ask gettext for
258 different queries depending on whether the numerical values in
259 question are 1, 2, more than 2, or in some cases 0, and you still haven\*(Aqt
260 figured out the problem with the different word order in Chinese.
261 .PP
262 Then your Russian translator calls on the phone, to \fIpersonally\fR tell
263 you the bad news about how really unpleasant your life is about to
264 become:
265 .PP
266 Russian, like German or Latin, is an inflectional language; that is, nouns
267 and adjectives have to take endings that depend on their case
268 (i.e., nominative, accusative, genitive, etc...) \-\- which is roughly a matter of
269 what role they have in syntax of the sentence \-\-
270 as well as on the grammatical gender (i.e., masculine, feminine, neuter)
271 and number (i.e., singular or plural) of the noun, as well as on the
272 declension class of the noun. But unlike with most other inflected languages,
273 putting a number\-phrase (like "ten" or "forty\-three", or their Arabic
274 numeral equivalents) in front of noun in Russian can change the case and
275 number that noun is, and therefore the endings you have to put on it.
276 .PP
277 He elaborates: In "I scanned \f(CW%g\fR directories", you\*(Aqd \fIexpect\fR
278 "directories" to be in the accusative case (since it is the direct
279 object in the sentence) and the plural number,
280 except where \f(CW$directory_count\fR is 1, then you\*(Aqd expect the singular, of
281 course. Just like Latin or German. \fIBut!\fR Where \f(CW$directory_count\fR %
282 10 is 1 ("%" for modulo, remember), assuming \f(CW$directory\fR count is an
283 integer, and except where \f(CW$directory_count\fR % 100 is 11, "directories"
284 is forced to become grammatically singular, which means it gets the
285 ending for the accusative singular... You begin to visualize the code
286 it\*(Aqd take to test for the problem so far, \fIand still work for Chinese
287 and Arabic and Italian\fR, and how many gettext items that\*(Aqd take, but
288 he keeps going... But where \f(CW$directory_count\fR % 10 is 2, 3, or 4
289 (except where \f(CW$directory_count\fR % 100 is 12, 13, or 14), the word for
290 "directories" is forced to be genitive singular \-\- which means another
291 ending... The room begins to spin around you, slowly at first... But
292 with \fIall other\fR integer values, since "directory" is an inanimate
293 noun, when preceded by a number and in the nominative or accusative
294 cases (as it is here, just your luck!), it does stay plural, but it is
295 forced into the genitive case \-\- yet another ending... And
296 you never hear him get to the part about how you\*(Aqre going to run into
297 similar (but maybe subtly different) problems with other Slavic
298 languages like Polish, because the floor comes up to meet you, and you
299 fade into unconsciousness.
300 .PP
301 The above cautionary tale relates how an attempt at localization can
302 lead from programmer consternation, to program obfuscation, to a need
303 for sedation. But careful evaluation shows that your choice of tools
304 merely needed further consideration.
305 .SS "The Linguistic View"
306 .IX Subsection "The Linguistic View"
307 .RS 4
308 "It is more complicated than you think."
309 .Sp
310 \&\-\- The Eighth Networking Truth, from RFC 1925
311 .RE
312 .PP
313 The field of Linguistics has expended a great deal of effort over the
314 past century trying to find grammatical patterns which hold across
315 languages; it\*(Aqs been a constant process
316 of people making generalizations that should apply to all languages,
317 only to find out that, all too often, these generalizations fail \-\-
318 sometimes failing for just a few languages, sometimes whole classes of
319 languages, and sometimes nearly every language in the world except
320 English. Broad statistical trends are evident in what the "average
321 language" is like as far as what its rules can look like, must look
322 like, and cannot look like. But the "average language" is just as
323 unreal a concept as the "average person" \-\- it runs up against the
324 fact no language (or person) is, in fact, average. The wisdom of past
325 experience leads us to believe that any given language can do whatever
326 it wants, in any order, with appeal to any kind of grammatical
327 categories wants \-\- case, number, tense, real or metaphoric
328 characteristics of the things that words refer to, arbitrary or
329 predictable classifications of words based on what endings or prefixes
330 they can take, degree or means of certainty about the truth of
331 statements expressed, and so on, ad infinitum.
332 .PP
333 Mercifully, most localization tasks are a matter of finding ways to
334 translate whole phrases, generally sentences, where the context is
335 relatively set, and where the only variation in content is \fIusually\fR
336 in a number being expressed \-\- as in the example sentences above.
337 Translating specific, fully\-formed sentences is, in practice, fairly
338 foolproof \-\- which is good, because that\*(Aqs what\*(Aqs in the phrasebooks
339 that so many tourists rely on. Now, a given phrase (whether in a
340 phrasebook or in a gettext lexicon) in one language \fImight\fR have a
341 greater or lesser applicability than that phrase\*(Aqs translation into
342 another language \-\- for example, strictly speaking, in Arabic, the
343 "your" in "Your query matched..." would take a different form
344 depending on whether the user is male or female; so the Arabic
345 translation "your[feminine] query" is applicable in fewer cases than
346 the corresponding English phrase, which doesn\*(Aqt distinguish the user\*(Aqs
347 gender. (In practice, it\*(Aqs not feasible to have a program know the
348 user\*(Aqs gender, so the masculine "you" in Arabic is usually used, by
349 default.)
350 .PP
351 But in general, such surprises are rare when entire sentences are
352 being translated, especially when the functional context is restricted
353 to that of a computer interacting with a user either to convey a fact
354 or to prompt for a piece of information. So, for purposes of
355 localization, translation by phrase (generally by sentence) is both the
356 simplest and the least problematic.
357 .SS "Breaking gettext"
358 .IX Subsection "Breaking gettext"
359 .RS 4
360 "It Has To Work."
361 .Sp
362 \&\-\- First Networking Truth, RFC 1925
363 .RE
364 .PP
365 Consider that sentences in a tourist phrasebook are of two types: ones
366 like "How do I get to the marketplace?" that don\*(Aqt have any blanks to
367 fill in, and ones like "How much do these _\|_\|_ cost?", where there\*(Aqs
368 one or more blanks to fill in (and these are usually linked to a
369 list of words that you can put in that blank: "fish", "potatoes",
370 "tomatoes", etc.). The ones with no blanks are no problem, but the
371 fill\-in\-the\-blank ones may not be really straightforward. If it\*(Aqs a
372 Swahili phrasebook, for example, the authors probably didn\*(Aqt bother to
373 tell you the complicated ways that the verb "cost" changes its
374 inflectional prefix depending on the noun you\*(Aqre putting in the blank.
375 The trader in the marketplace will still understand what you\*(Aqre saying if
376 you say "how much do these potatoes cost?" with the wrong
377 inflectional prefix on "cost". After all, \fIyou\fR can\*(Aqt speak proper Swahili,
378 \&\fIyou\*(Aqre\fR just a tourist. But while tourists can be stupid, computers
379 are supposed to be smart; the computer should be able to fill in the
380 blank, and still have the results be grammatical.
381 .PP
382 In other words, a phrasebook entry takes some values as parameters
383 (the things that you fill in the blank or blanks), and provides a value
384 based on these parameters, where the way you get that final value from
385 the given values can, properly speaking, involve an arbitrarily
386 complex series of operations. (In the case of Chinese, it\*(Aqd be not at
387 all complex, at least in cases like the examples at the beginning of
388 this article; whereas in the case of Russian it\*(Aqd be a rather complex
389 series of operations. And in some languages, the
390 complexity could be spread around differently: while the act of
391 putting a number\-expression in front of a noun phrase might not be
392 complex by itself, it may change how you have to, for example, inflect
393 a verb elsewhere in the sentence. This is what in syntax is called
394 "long\-distance dependencies".)
395 .PP
396 This talk of parameters and arbitrary complexity is just another way
397 to say that an entry in a phrasebook is what in a programming language
398 would be called a "function". Just so you don\*(Aqt miss it, this is the
399 crux of this article: \fIA phrase is a function; a phrasebook is a
400 bunch of functions.\fR
401 .PP
402 The reason that using gettext runs into walls (as in the above
403 second\-person horror story) is that you\*(Aqre trying to use a string (or
404 worse, a choice among a bunch of strings) to do what you really need a
405 function for \-\- which is futile. Preforming (s)printf interpolation
406 on the strings which you get back from gettext does allow you to do \fIsome\fR
407 common things passably well... sometimes... sort of; but, to paraphrase
408 what some people say about \f(CW\*(C`csh\*(C'\fR script programming, "it fools you
409 into thinking you can use it for real things, but you can\*(Aqt, and you
410 don\*(Aqt discover this until you\*(Aqve already spent too much time trying,
411 and by then it\*(Aqs too late."
412 .SS "Replacing gettext"
413 .IX Subsection "Replacing gettext"
414 So, what needs to replace gettext is a system that supports lexicons
415 of functions instead of lexicons of strings. An entry in a lexicon
416 from such a system should \fInot\fR look like this:
417 .PP
418 .Vb 1
419 \& "J\*(Aqai trouv\exE9 %g fichiers dans %g r\exE9pertoires"
420 .Ve
421 .PP
422 [\exE9 is e\-acute in Latin\-1. Some pod renderers would
423 scream if I used the actual character here. \-\- SB]
424 .PP
425 but instead like this, bearing in mind that this is just a first stab:
426 .PP
427 .Vb 8
428 \& sub I_found_X1_files_in_X2_directories {
429 \& my( $files, $dirs ) = @_[0,1];
430 \& $files = sprintf("%g %s", $files,
431 \& $files == 1 ? \*(Aqfichier\*(Aq : \*(Aqfichiers\*(Aq);
432 \& $dirs = sprintf("%g %s", $dirs,
433 \& $dirs == 1 ? "r\exE9pertoire" : "r\exE9pertoires");
434 \& return "J\*(Aqai trouv\exE9 $files dans $dirs.";
435 \& }
436 .Ve
437 .PP
438 Now, there\*(Aqs no particularly obvious way to store anything but strings
439 in a gettext lexicon; so it looks like we just have to start over and
440 make something better, from scratch. I call my shot at a
441 gettext\-replacement system "Maketext", or, in CPAN terms,
442 Locale::Maketext.
443 .PP
444 When designing Maketext, I chose to plan its main features in terms of
445 "buzzword compliance". And here are the buzzwords:
446 .SS "Buzzwords: Abstraction and Encapsulation"
447 .IX Subsection "Buzzwords: Abstraction and Encapsulation"
448 The complexity of the language you\*(Aqre trying to output a phrase in is
449 entirely abstracted inside (and encapsulated within) the Maketext module
450 for that interface. When you call:
451 .PP
452 .Vb 2
453 \& print $lang\->maketext("You have [quant,_1,piece] of new mail.",
454 \& scalar(@messages));
455 .Ve
456 .PP
457 you don\*(Aqt know (and in fact can\*(Aqt easily find out) whether this will
458 involve lots of figuring, as in Russian (if \f(CW$lang\fR is a handle to the
459 Russian module), or relatively little, as in Chinese. That kind of
460 abstraction and encapsulation may encourage other pleasant buzzwords
461 like modularization and stratification, depending on what design
462 decisions you make.
463 .SS "Buzzword: Isomorphism"
464 .IX Subsection "Buzzword: Isomorphism"
465 "Isomorphism" means "having the same structure or form"; in discussions
466 of program design, the word takes on the special, specific meaning that
467 your implementation of a solution to a problem \fIhas the same
468 structure\fR as, say, an informal verbal description of the solution, or
469 maybe of the problem itself. Isomorphism is, all things considered,
470 a good thing \-\- it\*(Aqs what problem\-solving (and solution\-implementing)
471 should look like.
472 .PP
473 What\*(Aqs wrong the with gettext\-using code like this...
474 .PP
475 .Vb 9
476 \& printf( $file_count == 1 ?
477 \& ( $directory_count == 1 ?
478 \& "Your query matched %g file in %g directory." :
479 \& "Your query matched %g file in %g directories." ) :
480 \& ( $directory_count == 1 ?
481 \& "Your query matched %g files in %g directory." :
482 \& "Your query matched %g files in %g directories." ),
483 \& $file_count, $directory_count,
484 \& );
485 .Ve
486 .PP
487 is first off that it\*(Aqs not well abstracted \-\- these ways of testing
488 for grammatical number (as in the expressions like \f(CW\*(C`foo == 1 ?
489 singular_form : plural_form\*(C'\fR) should be abstracted to each language
490 module, since how you get grammatical number is language\-specific.
491 .PP
492 But second off, it\*(Aqs not isomorphic \-\- the "solution" (i.e., the
493 phrasebook entries) for Chinese maps from these four English phrases to
494 the one Chinese phrase that fits for all of them. In other words, the
495 informal solution would be "The way to say what you want in Chinese is
496 with the one phrase \*(AqFor your question, in Y directories you would
497 find X files\*(Aq" \-\- and so the implemented solution should be,
498 isomorphically, just a straightforward way to spit out that one
499 phrase, with numerals properly interpolated. It shouldn\*(Aqt have to map
500 from the complexity of other languages to the simplicity of this one.
501 .SS "Buzzword: Inheritance"
502 .IX Subsection "Buzzword: Inheritance"
503 There\*(Aqs a great deal of reuse possible for sharing of phrases between
504 modules for related dialects, or for sharing of auxiliary functions
505 between related languages. (By "auxiliary functions", I mean
506 functions that don\*(Aqt produce phrase\-text, but which, say, return an
507 answer to "does this number require a plural noun after it?". Such
508 auxiliary functions would be used in the internal logic of functions
509 that actually do produce phrase\-text.)
510 .PP
511 In the case of sharing phrases, consider that you have an interface
512 already localized for American English (probably by having been
513 written with that as the native locale, but that\*(Aqs incidental).
514 Localizing it for UK English should, in practical terms, be just a
515 matter of running it past a British person with the instructions to
516 indicate what few phrases would benefit from a change in spelling or
517 possibly minor rewording. In that case, you should be able to put in
518 the UK English localization module \fIonly\fR those phrases that are
519 UK\-specific, and for all the rest, \fIinherit\fR from the American
520 English module. (And I expect this same situation would apply with
521 Brazilian and Continental Portugese, possibly with some \fIvery\fR
522 closely related languages like Czech and Slovak, and possibly with the
523 slightly different "versions" of written Mandarin Chinese, as I hear exist in
524 Taiwan and mainland China.)
525 .PP
526 As to sharing of auxiliary functions, consider the problem of Russian
527 numbers from the beginning of this article; obviously, you\*(Aqd want to
528 write only once the hairy code that, given a numeric value, would
529 return some specification of which case and number a given quantified
530 noun should use. But suppose that you discover, while localizing an
531 interface for, say, Ukrainian (a Slavic language related to Russian,
532 spoken by several million people, many of whom would be relieved to
533 find that your Web site\*(Aqs or software\*(Aqs interface is available in
534 their language), that the rules in Ukrainian are the same as in Russian
535 for quantification, and probably for many other grammatical functions.
536 While there may well be no phrases in common between Russian and
537 Ukrainian, you could still choose to have the Ukrainian module inherit
538 from the Russian module, just for the sake of inheriting all the
539 various grammatical methods. Or, probably better organizationally,
540 you could move those functions to a module called \f(CW\*(C`_E_Slavic\*(C'\fR or
541 something, which Russian and Ukrainian could inherit useful functions
542 from, but which would (presumably) provide no lexicon.
543 .SS "Buzzword: Concision"
544 .IX Subsection "Buzzword: Concision"
545 Okay, concision isn\*(Aqt a buzzword. But it should be, so I decree that
546 as a new buzzword, "concision" means that simple common things should
547 be expressible in very few lines (or maybe even just a few characters)
548 of code \-\- call it a special case of "making simple things easy and
549 hard things possible", and see also the role it played in the
550 MIDI::Simple language, discussed elsewhere in this issue [TPJ#13].
551 .PP
552 Consider our first stab at an entry in our "phrasebook of functions":
553 .PP
554 .Vb 8
555 \& sub I_found_X1_files_in_X2_directories {
556 \& my( $files, $dirs ) = @_[0,1];
557 \& $files = sprintf("%g %s", $files,
558 \& $files == 1 ? \*(Aqfichier\*(Aq : \*(Aqfichiers\*(Aq);
559 \& $dirs = sprintf("%g %s", $dirs,
560 \& $dirs == 1 ? "r\exE9pertoire" : "r\exE9pertoires");
561 \& return "J\*(Aqai trouv\exE9 $files dans $dirs.";
562 \& }
563 .Ve
564 .PP
565 You may sense that a lexicon (to use a non\-committal catch\-all term for a
566 collection of things you know how to say, regardless of whether they\*(Aqre
567 phrases or words) consisting of functions \fIexpressed\fR as above would
568 make for rather long\-winded and repetitive code \-\- even if you wisely
569 rewrote this to have quantification (as we call adding a number
570 expression to a noun phrase) be a function called like:
571 .PP
572 .Vb 6
573 \& sub I_found_X1_files_in_X2_directories {
574 \& my( $files, $dirs ) = @_[0,1];
575 \& $files = quant($files, "fichier");
576 \& $dirs = quant($dirs, "r\exE9pertoire");
577 \& return "J\*(Aqai trouv\exE9 $files dans $dirs.";
578 \& }
579 .Ve
580 .PP
581 And you may also sense that you do not want to bother your translators
582 with having to write Perl code \-\- you\*(Aqd much rather that they spend
583 their \fIvery costly time\fR on just translation. And this is to say
584 nothing of the near impossibility of finding a commercial translator
585 who would know even simple Perl.
586 .PP
587 In a first\-hack implementation of Maketext, each language\-module\*(Aqs
588 lexicon looked like this:
589 .PP
590 .Vb 10
591 \& %Lexicon = (
592 \& "I found %g files in %g directories"
593 \& => sub {
594 \& my( $files, $dirs ) = @_[0,1];
595 \& $files = quant($files, "fichier");
596 \& $dirs = quant($dirs, "r\exE9pertoire");
597 \& return "J\*(Aqai trouv\exE9 $files dans $dirs.";
598 \& },
599 \& ... and so on with other phrase => sub mappings ...
600 \& );
601 .Ve
602 .PP
603 but I immediately went looking for some more concise way to basically
604 denote the same phrase\-function \-\- a way that would also serve to
605 concisely denote \fImost\fR phrase\-functions in the lexicon for \fImost\fR
606 languages. After much time and even some actual thought, I decided on
607 this system:
608 .PP
609 * Where a value in a \f(CW%Lexicon\fR hash is a contentful string instead of
610 an anonymous sub (or, conceivably, a coderef), it would be interpreted
611 as a sort of shorthand expression of what the sub does. When accessed
612 for the first time in a session, it is parsed, turned into Perl code,
613 and then eval\*(Aqd into an anonymous sub; then that sub replaces the
614 original string in that lexicon. (That way, the work of parsing and
615 evaling the shorthand form for a given phrase is done no more than
616 once per session.)
617 .PP
618 * Calls to \f(CW\*(C`maketext\*(C'\fR (as Maketext\*(Aqs main function is called) happen
619 thru a "language session handle", notionally very much like an IO
620 handle, in that you open one at the start of the session, and use it
621 for "sending signals" to an object in order to have it return the text
622 you want.
623 .PP
624 So, this:
625 .PP
626 .Vb 2
627 \& $lang\->maketext("You have [quant,_1,piece] of new mail.",
628 \& scalar(@messages));
629 .Ve
630 .PP
631 basically means this: look in the lexicon for \f(CW$lang\fR (which may inherit
632 from any number of other lexicons), and find the function that we
633 happen to associate with the string "You have [quant,_1,piece] of new
634 mail" (which is, and should be, a functioning "shorthand" for this
635 function in the native locale \-\- English in this case). If you find
636 such a function, call it with \f(CW$lang\fR as its first parameter (as if it
637 were a method), and then a copy of scalar(@messages) as its second,
638 and then return that value. If that function was found, but was in
639 string shorthand instead of being a fully specified function, parse it
640 and make it into a function before calling it the first time.
641 .PP
642 * The shorthand uses code in brackets to indicate method calls that
643 should be performed. A full explanation is not in order here, but a
644 few examples will suffice:
645 .PP
646 .Vb 1
647 \& "You have [quant,_1,piece] of new mail."
648 .Ve
649 .PP
650 The above code is shorthand for, and will be interpreted as,
651 this:
652 .PP
653 .Vb 8
654 \& sub {
655 \& my $handle = $_[0];
656 \& my(@params) = @_;
657 \& return join \*(Aq\*(Aq,
658 \& "You have ",
659 \& $handle\->quant($params[1], \*(Aqpiece\*(Aq),
660 \& "of new mail.";
661 \& }
662 .Ve
663 .PP
664 where "quant" is the name of a method you\*(Aqre using to quantify the
665 noun "piece" with the number \f(CW$params\fR[0].
666 .PP
667 A string with no brackety calls, like this:
668 .PP
669 .Vb 1
670 \& "Your search expression was malformed."
671 .Ve
672 .PP
673 is somewhat of a degenerate case, and just gets turned into:
674 .PP
675 .Vb 1
676 \& sub { return "Your search expression was malformed." }
677 .Ve
678 .PP
679 However, not everything you can write in Perl code can be written in
680 the above shorthand system \-\- not by a long shot. For example, consider
681 the Italian translator from the beginning of this article, who wanted
682 the Italian for "I didn\*(Aqt find any files" as a special case, instead
683 of "I found 0 files". That couldn\*(Aqt be specified (at least not easily
684 or simply) in our shorthand system, and it would have to be written
685 out in full, like this:
686 .PP
687 .Vb 10
688 \& sub { # pretend the English strings are in Italian
689 \& my($handle, $files, $dirs) = @_[0,1,2];
690 \& return "I didn\*(Aqt find any files" unless $files;
691 \& return join \*(Aq\*(Aq,
692 \& "I found ",
693 \& $handle\->quant($files, \*(Aqfile\*(Aq),
694 \& " in ",
695 \& $handle\->quant($dirs, \*(Aqdirectory\*(Aq),
696 \& ".";
697 \& }
698 .Ve
699 .PP
700 Next to a lexicon full of shorthand code, that sort of sticks out like a
701 sore thumb \-\- but this \fIis\fR a special case, after all; and at least
702 it\*(Aqs possible, if not as concise as usual.
703 .PP
704 As to how you\*(Aqd implement the Russian example from the beginning of
705 the article, well, There\*(Aqs More Than One Way To Do It, but it could be
706 something like this (using English words for Russian, just so you know
707 what\*(Aqs going on):
708 .PP
709 .Vb 1
710 \& "I [quant,_1,directory,accusative] scanned."
711 .Ve
712 .PP
713 This shifts the burden of complexity off to the quant method. That
714 method\*(Aqs parameters are: the numeric value it\*(Aqs going to use to
715 quantify something; the Russian word it\*(Aqs going to quantify; and the
716 parameter "accusative", which you\*(Aqre using to mean that this
717 sentence\*(Aqs syntax wants a noun in the accusative case there, although
718 that quantification method may have to overrule, for grammatical
719 reasons you may recall from the beginning of this article.
720 .PP
721 Now, the Russian quant method here is responsible not only for
722 implementing the strange logic necessary for figuring out how Russian
723 number\-phrases impose case and number on their noun\-phrases, but also
724 for inflecting the Russian word for "directory". How that inflection
725 is to be carried out is no small issue, and among the solutions I\*(Aqve
726 seen, some (like variations on a simple lookup in a hash where all
727 possible forms are provided for all necessary words) are
728 straightforward but \fIcan\fR become cumbersome when you need to inflect
729 more than a few dozen words; and other solutions (like using
730 algorithms to model the inflections, storing only root forms and
731 irregularities) \fIcan\fR involve more overhead than is justifiable for
732 all but the largest lexicons.
733 .PP
734 Mercifully, this design decision becomes crucial only in the hairiest
735 of inflected languages, of which Russian is by no means the \fIworst\fR case
736 scenario, but is worse than most. Most languages have simpler
737 inflection systems; for example, in English or Swahili, there are
738 generally no more than two possible inflected forms for a given noun
739 ("error/errors"; "kosa/makosa"), and the
740 rules for producing these forms are fairly simple \-\- or at least,
741 simple rules can be formulated that work for most words, and you can
742 then treat the exceptions as just "irregular", at least relative to
743 your ad hoc rules. A simpler inflection system (simpler rules, fewer
744 forms) means that design decisions are less crucial to maintaining
745 sanity, whereas the same decisions could incur
746 overhead\-versus\-scalability problems in languages like Russian. It
747 may \fIalso\fR be likely that code (possibly in Perl, as with
748 Lingua::EN::Inflect, for English nouns) has already
749 been written for the language in question, whether simple or complex.
750 .PP
751 Moreover, a third possibility may even be simpler than anything
752 discussed above: "Just require that all possible (or at least
753 applicable) forms be provided in the call to the given language\*(Aqs quant
754 method, as in:"
755 .PP
756 .Vb 1
757 \& "I found [quant,_1,file,files]."
758 .Ve
759 .PP
760 That way, quant just has to chose which form it needs, without having
761 to look up or generate anything. While possibly not optimal for
762 Russian, this should work well for most other languages, where
763 quantification is not as complicated an operation.
764 .SS "The Devil in the Details"
765 .IX Subsection "The Devil in the Details"
766 There\*(Aqs plenty more to Maketext than described above \-\- for example,
767 there\*(Aqs the details of how language tags ("en\-US", "i\-pwn", "fi",
768 etc.) or locale IDs ("en_US") interact with actual module naming
769 ("BogoQuery/Locale/en_us.pm"), and what magic can ensue; there\*(Aqs the
770 details of how to record (and possibly negotiate) what character
771 encoding Maketext will return text in (UTF8? Latin\-1? KOI8?). There\*(Aqs
772 the interesting fact that Maketext is for localization, but nowhere
773 actually has a "\f(CW\*(C`use locale;\*(C'\fR" anywhere in it. For the curious,
774 there\*(Aqs the somewhat frightening details of how I actually
775 implement something like data inheritance so that searches across
776 modules\*(Aq \f(CW%Lexicon\fR hashes can parallel how Perl implements method
777 inheritance.
778 .PP
779 And, most importantly, there\*(Aqs all the practical details of how to
780 actually go about deriving from Maketext so you can use it for your
781 interfaces, and the various tools and conventions for starting out and
782 maintaining individual language modules.
783 .PP
784 That is all covered in the documentation for Locale::Maketext and the
785 modules that come with it, available in CPAN. After having read this
786 article, which covers the why\*(Aqs of Maketext, the documentation,
787 which covers the how\*(Aqs of it, should be quite straightforward.
788 .SS "The Proof in the Pudding: Localizing Web Sites"
789 .IX Subsection "The Proof in the Pudding: Localizing Web Sites"
790 Maketext and gettext have a notable difference: gettext is in C,
791 accessible thru C library calls, whereas Maketext is in Perl, and
792 really can\*(Aqt work without a Perl interpreter (although I suppose
793 something like it could be written for C). Accidents of history (and
794 not necessarily lucky ones) have made C++ the most common language for
795 the implementation of applications like word processors, Web browsers,
796 and even many in\-house applications like custom query systems. Current
797 conditions make it somewhat unlikely that the next one of any of these
798 kinds of applications will be written in Perl, albeit clearly more for
799 reasons of custom and inertia than out of consideration of what is the
800 right tool for the job.
801 .PP
802 However, other accidents of history have made Perl a well\-accepted
803 language for design of server\-side programs (generally in CGI form)
804 for Web site interfaces. Localization of static pages in Web sites is
805 trivial, feasible either with simple language\-negotiation features in
806 servers like Apache, or with some kind of server\-side inclusions of
807 language\-appropriate text into layout templates. However, I think
808 that the localization of Perl\-based search systems (or other kinds of
809 dynamic content) in Web sites, be they public or access\-restricted,
810 is where Maketext will see the greatest use.
811 .PP
812 I presume that it would be only the exceptional Web site that gets
813 localized for English \fIand\fR Chinese \fIand\fR Italian \fIand\fR Arabic
814 \&\fIand\fR Russian, to recall the languages from the beginning of this
815 article \-\- to say nothing of German, Spanish, French, Japanese,
816 Finnish, and Hindi, to name a few languages that benefit from large
817 numbers of programmers or Web viewers or both.
818 .PP
819 However, the ever\-increasing internationalization of the Web (whether
820 measured in terms of amount of content, of numbers of content writers
821 or programmers, or of size of content audiences) makes it increasingly
822 likely that the interface to the average Web\-based dynamic content
823 service will be localized for two or maybe three languages. It is my
824 hope that Maketext will make that task as simple as possible, and will
825 remove previous barriers to localization for languages dissimilar to
826 English.
827 .PP
828 .Vb 1
829 \& _\|_END_\|_
830 .Ve
831 .PP
832 Sean M. Burke ([email protected]) has a Master\*(Aqs in linguistics
833 from Northwestern University; he specializes in language technology.
834 Jordan Lachler ([email protected]) is a PhD student in the Department of
835 Linguistics at the University of New Mexico; he specializes in
836 morphology and pedagogy of North American native languages.
837 .SS References
838 .IX Subsection "References"
839 Alvestrand, Harald Tveit. 1995. \fIRFC 1766: Tags for the
840 Identification of Languages.\fR
841 \&\f(CW\*(C`<http://www.ietf.org/rfc/rfc1766.txt>\*(C'\fR
842 [Now see RFC 3066.]
843 .PP
844 Callon, Ross, editor. 1996. \fIRFC 1925: The Twelve
845 Networking Truths.\fR
846 \&\f(CW\*(C`<http://www.ietf.org/rfc/rfc1925.txt>\*(C'\fR
847 .PP
848 Drepper, Ulrich, Peter Miller,
849 and François Pinard. 1995\-2001. GNU
850 \&\f(CW\*(C`gettext\*(C'\fR. Available in \f(CW\*(C`<ftp://prep.ai.mit.edu/pub/gnu/>\*(C'\fR, with
851 extensive docs in the distribution tarball. [Since
852 I wrote this article in 1998, I now see that the
853 gettext docs are now trying more to come to terms with
854 plurality. Whether useful conclusions have come from it
855 is another question altogether. \-\- SMB, May 2001]
856 .PP
857 Forbes, Nevill. 1964. \fIRussian Grammar.\fR Third Edition, revised
858 by J. C. Dumbreck. Oxford University Press.