Debian Codegraph @

upstream/debian-unstable/man3/Locale::Maketext::TPJ13.3perl

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "Locale::Maketext::TPJ13 3perl"
58 .TH Locale::Maketext::TPJ13 3perl 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 Locale::Maketext::TPJ13 \-\- article about software localization
65 .SH SYNOPSIS
66 .IX Header "SYNOPSIS"
67 .Vb 1
68 \& # This an article, not a module.
69 .Ve
70 .SH DESCRIPTION
71 .IX Header "DESCRIPTION"
72 The following article by Sean M. Burke and Jordan Lachler
73 first appeared in \fIThe Perl Journal\fR #13
74 and is copyright 1999 The Perl Journal. It appears
75 courtesy of Jon Orwant and The Perl Journal. This document may be
76 distributed under the same terms as Perl itself.
77 .SH "Localization and Perl: gettext breaks, Maketext fixes"
78 .IX Header "Localization and Perl: gettext breaks, Maketext fixes"
79 by Sean M. Burke and Jordan Lachler
80 .PP
81 This article points out cases where gettext (a common system for
82 localizing software interfaces \-\- i.e., making them work in the user's
83 language of choice) fails because of basic differences between human
84 languages. This article then describes Maketext, a new system capable
85 of correctly treating these differences.
86 .SS "A Localization Horror Story: It Could Happen To You"
87 .IX Subsection "A Localization Horror Story: It Could Happen To You"
88 .RS 4
89 "There are a number of languages spoken by human beings in this
90 world."
91 .Sp
92 \&\-\- Harald Tveit Alvestrand, in RFC 1766, "Tags for the
93 Identification of Languages"
94 .RE
95 .PP
96 Imagine that your task for the day is to localize a piece of software
97 \&\-\- and luckily for you, the only output the program emits is two
98 messages, like this:
99 .PP
100 .Vb 1
101 \& I scanned 12 directories.
102 \&
103 \& Your query matched 10 files in 4 directories.
104 .Ve
105 .PP
106 So how hard could that be? You look at the code that
107 produces the first item, and it reads:
108 .PP
109 .Vb 2
110 \& printf("I scanned %g directories.",
111 \& $directory_count);
112 .Ve
113 .PP
114 You think about that, and realize that it doesn't even work right for
115 English, as it can produce this output:
116 .PP
117 .Vb 1
118 \& I scanned 1 directories.
119 .Ve
120 .PP
121 So you rewrite it to read:
122 .PP
123 .Vb 5
124 \& printf("I scanned %g %s.",
125 \& $directory_count,
126 \& $directory_count == 1 ?
127 \& "directory" : "directories",
128 \& );
129 .Ve
130 .PP
131 \&...which does the Right Thing. (In case you don't recall, "%g" is for
132 locale-specific number interpolation, and "%s" is for string
133 interpolation.)
134 .PP
135 But you still have to localize it for all the languages you're
136 producing this software for, so you pull Locale::gettext off of CPAN
137 so you can access the \f(CW\*(C`gettext\*(C'\fR C functions you've heard are standard
138 for localization tasks.
139 .PP
140 And you write:
141 .PP
142 .Vb 5
143 \& printf(gettext("I scanned %g %s."),
144 \& $dir_scan_count,
145 \& $dir_scan_count == 1 ?
146 \& gettext("directory") : gettext("directories"),
147 \& );
148 .Ve
149 .PP
150 But you then read in the gettext manual (Drepper, Miller, and Pinard 1995)
151 that this is not a good idea, since how a single word like "directory"
152 or "directories" is translated may depend on context \-\- and this is
153 true, since in a case language like German or Russian, you'd may need
154 these words with a different case ending in the first instance (where the
155 word is the object of a verb) than in the second instance, which you haven't even
156 gotten to yet (where the word is the object of a preposition, "in \f(CW%g\fR
157 directories") \-\- assuming these keep the same syntax when translated
158 into those languages.
159 .PP
160 So, on the advice of the gettext manual, you rewrite:
161 .PP
162 .Vb 4
163 \& printf( $dir_scan_count == 1 ?
164 \& gettext("I scanned %g directory.") :
165 \& gettext("I scanned %g directories."),
166 \& $dir_scan_count );
167 .Ve
168 .PP
169 So, you email your various translators (the boss decides that the
170 languages du jour are Chinese, Arabic, Russian, and Italian, so you
171 have one translator for each), asking for translations for "I scanned
172 \&\f(CW%g\fR directory." and "I scanned \f(CW%g\fR directories.". When they reply,
173 you'll put that in the lexicons for gettext to use when it localizes
174 your software, so that when the user is running under the "zh"
175 (Chinese) locale, gettext("I scanned \f(CW%g\fR directory.") will return the
176 appropriate Chinese text, with a "%g" in there where printf can then
177 interpolate \f(CW$dir_scan\fR.
178 .PP
179 Your Chinese translator emails right back \-\- he says both of these
180 phrases translate to the same thing in Chinese, because, in linguistic
181 jargon, Chinese "doesn't have number as a grammatical category" \-\-
182 whereas English does. That is, English has grammatical rules that
183 refer to "number", i.e., whether something is grammatically singular
184 or plural; and one of these rules is the one that forces nouns to take
185 a plural suffix (generally "s") when in a plural context, as they are when
186 they follow a number other than "one" (including, oddly enough, "zero").
187 Chinese has no such rules, and so has just the one phrase where English
188 has two. But, no problem, you can have this one Chinese phrase appear
189 as the translation for the two English phrases in the "zh" gettext
190 lexicon for your program.
191 .PP
192 Emboldened by this, you dive into the second phrase that your software
193 needs to output: "Your query matched 10 files in 4 directories.". You notice
194 that if you want to treat phrases as indivisible, as the gettext
195 manual wisely advises, you need four cases now, instead of two, to
196 cover the permutations of singular and plural on the two items,
197 \&\f(CW$dir_count\fR and \f(CW$file_count\fR. So you try this:
198 .PP
199 .Vb 9
200 \& printf( $file_count == 1 ?
201 \& ( $directory_count == 1 ?
202 \& gettext("Your query matched %g file in %g directory.") :
203 \& gettext("Your query matched %g file in %g directories.") ) :
204 \& ( $directory_count == 1 ?
205 \& gettext("Your query matched %g files in %g directory.") :
206 \& gettext("Your query matched %g files in %g directories.") ),
207 \& $file_count, $directory_count,
208 \& );
209 .Ve
210 .PP
211 (The case of "1 file in 2 [or more] directories" could, I suppose,
212 occur in the case of symlinking or something of the sort.)
213 .PP
214 It occurs to you that this is not the prettiest code you've ever
215 written, but this seems the way to go. You mail off to the
216 translators asking for translations for these four cases. The
217 Chinese guy replies with the one phrase that these all translate to in
218 Chinese, and that phrase has two "%g"s in it, as it should \-\- but
219 there's a problem. He translates it word-for-word back: "In \f(CW%g\fR
220 directories contains \f(CW%g\fR files match your query." The \f(CW%g\fR
221 slots are in an order reverse to what they are in English. You wonder
222 how you'll get gettext to handle that.
223 .PP
224 But you put it aside for the moment, and optimistically hope that the
225 other translators won't have this problem, and that their languages
226 will be better behaved \-\- i.e., that they will be just like English.
227 .PP
228 But the Arabic translator is the next to write back. First off, your
229 code for "I scanned \f(CW%g\fR directory." or "I scanned \f(CW%g\fR directories."
230 assumes there's only singular or plural. But, to use linguistic
231 jargon again, Arabic has grammatical number, like English (but unlike
232 Chinese), but it's a three-term category: singular, dual, and plural.
233 In other words, the way you say "directory" depends on whether there's
234 one directory, or \fItwo\fR of them, or \fImore than two\fR of them. Your
235 test of \f(CW\*(C`($directory == 1)\*(C'\fR no longer does the job. And it means
236 that where English's grammatical category of number necessitates
237 only the two permutations of the first sentence based on "directory
238 [singular]" and "directories [plural]", Arabic has three \-\- and,
239 worse, in the second sentence ("Your query matched \f(CW%g\fR file in \f(CW%g\fR
240 directory."), where English has four, Arabic has nine. You sense
241 an unwelcome, exponential trend taking shape.
242 .PP
243 Your Italian translator emails you back and says that "I searched 0
244 directories" (a possible English output of your program) is stilted,
245 and if you think that's fine English, that's your problem, but that
246 \&\fIjust will not do\fR in the language of Dante. He insists that where
247 \&\f(CW$directory_count\fR is 0, your program should produce the Italian text
248 for "I \fIdidn't\fR scan \fIany\fR directories.". And ditto for "I didn't
249 match any files in any directories", although he says the last part
250 about "in any directories" should probably just be left off.
251 .PP
252 You wonder how you'll get gettext to handle this; to accommodate the
253 ways Arabic, Chinese, and Italian deal with numbers in just these few
254 very simple phrases, you need to write code that will ask gettext for
255 different queries depending on whether the numerical values in
256 question are 1, 2, more than 2, or in some cases 0, and you still haven't
257 figured out the problem with the different word order in Chinese.
258 .PP
259 Then your Russian translator calls on the phone, to \fIpersonally\fR tell
260 you the bad news about how really unpleasant your life is about to
261 become:
262 .PP
263 Russian, like German or Latin, is an inflectional language; that is, nouns
264 and adjectives have to take endings that depend on their case
265 (i.e., nominative, accusative, genitive, etc...) \-\- which is roughly a matter of
266 what role they have in syntax of the sentence \-\-
267 as well as on the grammatical gender (i.e., masculine, feminine, neuter)
268 and number (i.e., singular or plural) of the noun, as well as on the
269 declension class of the noun. But unlike with most other inflected languages,
270 putting a number-phrase (like "ten" or "forty-three", or their Arabic
271 numeral equivalents) in front of noun in Russian can change the case and
272 number that noun is, and therefore the endings you have to put on it.
273 .PP
274 He elaborates: In "I scanned \f(CW%g\fR directories", you'd \fIexpect\fR
275 "directories" to be in the accusative case (since it is the direct
276 object in the sentence) and the plural number,
277 except where \f(CW$directory_count\fR is 1, then you'd expect the singular, of
278 course. Just like Latin or German. \fIBut!\fR Where \f(CW$directory_count\fR %
279 10 is 1 ("%" for modulo, remember), assuming \f(CW$directory\fR count is an
280 integer, and except where \f(CW$directory_count\fR % 100 is 11, "directories"
281 is forced to become grammatically singular, which means it gets the
282 ending for the accusative singular... You begin to visualize the code
283 it'd take to test for the problem so far, \fIand still work for Chinese
284 and Arabic and Italian\fR, and how many gettext items that'd take, but
285 he keeps going... But where \f(CW$directory_count\fR % 10 is 2, 3, or 4
286 (except where \f(CW$directory_count\fR % 100 is 12, 13, or 14), the word for
287 "directories" is forced to be genitive singular \-\- which means another
288 ending... The room begins to spin around you, slowly at first... But
289 with \fIall other\fR integer values, since "directory" is an inanimate
290 noun, when preceded by a number and in the nominative or accusative
291 cases (as it is here, just your luck!), it does stay plural, but it is
292 forced into the genitive case \-\- yet another ending... And
293 you never hear him get to the part about how you're going to run into
294 similar (but maybe subtly different) problems with other Slavic
295 languages like Polish, because the floor comes up to meet you, and you
296 fade into unconsciousness.
297 .PP
298 The above cautionary tale relates how an attempt at localization can
299 lead from programmer consternation, to program obfuscation, to a need
300 for sedation. But careful evaluation shows that your choice of tools
301 merely needed further consideration.
302 .SS "The Linguistic View"
303 .IX Subsection "The Linguistic View"
304 .RS 4
305 "It is more complicated than you think."
306 .Sp
307 \&\-\- The Eighth Networking Truth, from RFC 1925
308 .RE
309 .PP
310 The field of Linguistics has expended a great deal of effort over the
311 past century trying to find grammatical patterns which hold across
312 languages; it's been a constant process
313 of people making generalizations that should apply to all languages,
314 only to find out that, all too often, these generalizations fail \-\-
315 sometimes failing for just a few languages, sometimes whole classes of
316 languages, and sometimes nearly every language in the world except
317 English. Broad statistical trends are evident in what the "average
318 language" is like as far as what its rules can look like, must look
319 like, and cannot look like. But the "average language" is just as
320 unreal a concept as the "average person" \-\- it runs up against the
321 fact no language (or person) is, in fact, average. The wisdom of past
322 experience leads us to believe that any given language can do whatever
323 it wants, in any order, with appeal to any kind of grammatical
324 categories wants \-\- case, number, tense, real or metaphoric
325 characteristics of the things that words refer to, arbitrary or
326 predictable classifications of words based on what endings or prefixes
327 they can take, degree or means of certainty about the truth of
328 statements expressed, and so on, ad infinitum.
329 .PP
330 Mercifully, most localization tasks are a matter of finding ways to
331 translate whole phrases, generally sentences, where the context is
332 relatively set, and where the only variation in content is \fIusually\fR
333 in a number being expressed \-\- as in the example sentences above.
334 Translating specific, fully-formed sentences is, in practice, fairly
335 foolproof \-\- which is good, because that's what's in the phrasebooks
336 that so many tourists rely on. Now, a given phrase (whether in a
337 phrasebook or in a gettext lexicon) in one language \fImight\fR have a
338 greater or lesser applicability than that phrase's translation into
339 another language \-\- for example, strictly speaking, in Arabic, the
340 "your" in "Your query matched..." would take a different form
341 depending on whether the user is male or female; so the Arabic
342 translation "your[feminine] query" is applicable in fewer cases than
343 the corresponding English phrase, which doesn't distinguish the user's
344 gender. (In practice, it's not feasible to have a program know the
345 user's gender, so the masculine "you" in Arabic is usually used, by
346 default.)
347 .PP
348 But in general, such surprises are rare when entire sentences are
349 being translated, especially when the functional context is restricted
350 to that of a computer interacting with a user either to convey a fact
351 or to prompt for a piece of information. So, for purposes of
352 localization, translation by phrase (generally by sentence) is both the
353 simplest and the least problematic.
354 .SS "Breaking gettext"
355 .IX Subsection "Breaking gettext"
356 .RS 4
357 "It Has To Work."
358 .Sp
359 \&\-\- First Networking Truth, RFC 1925
360 .RE
361 .PP
362 Consider that sentences in a tourist phrasebook are of two types: ones
363 like "How do I get to the marketplace?" that don't have any blanks to
364 fill in, and ones like "How much do these _\|_\|_ cost?", where there's
365 one or more blanks to fill in (and these are usually linked to a
366 list of words that you can put in that blank: "fish", "potatoes",
367 "tomatoes", etc.). The ones with no blanks are no problem, but the
368 fill-in-the-blank ones may not be really straightforward. If it's a
369 Swahili phrasebook, for example, the authors probably didn't bother to
370 tell you the complicated ways that the verb "cost" changes its
371 inflectional prefix depending on the noun you're putting in the blank.
372 The trader in the marketplace will still understand what you're saying if
373 you say "how much do these potatoes cost?" with the wrong
374 inflectional prefix on "cost". After all, \fIyou\fR can't speak proper Swahili,
375 \&\fIyou're\fR just a tourist. But while tourists can be stupid, computers
376 are supposed to be smart; the computer should be able to fill in the
377 blank, and still have the results be grammatical.
378 .PP
379 In other words, a phrasebook entry takes some values as parameters
380 (the things that you fill in the blank or blanks), and provides a value
381 based on these parameters, where the way you get that final value from
382 the given values can, properly speaking, involve an arbitrarily
383 complex series of operations. (In the case of Chinese, it'd be not at
384 all complex, at least in cases like the examples at the beginning of
385 this article; whereas in the case of Russian it'd be a rather complex
386 series of operations. And in some languages, the
387 complexity could be spread around differently: while the act of
388 putting a number-expression in front of a noun phrase might not be
389 complex by itself, it may change how you have to, for example, inflect
390 a verb elsewhere in the sentence. This is what in syntax is called
391 "long-distance dependencies".)
392 .PP
393 This talk of parameters and arbitrary complexity is just another way
394 to say that an entry in a phrasebook is what in a programming language
395 would be called a "function". Just so you don't miss it, this is the
396 crux of this article: \fIA phrase is a function; a phrasebook is a
397 bunch of functions.\fR
398 .PP
399 The reason that using gettext runs into walls (as in the above
400 second-person horror story) is that you're trying to use a string (or
401 worse, a choice among a bunch of strings) to do what you really need a
402 function for \-\- which is futile. Preforming (s)printf interpolation
403 on the strings which you get back from gettext does allow you to do \fIsome\fR
404 common things passably well... sometimes... sort of; but, to paraphrase
405 what some people say about \f(CW\*(C`csh\*(C'\fR script programming, "it fools you
406 into thinking you can use it for real things, but you can't, and you
407 don't discover this until you've already spent too much time trying,
408 and by then it's too late."
409 .SS "Replacing gettext"
410 .IX Subsection "Replacing gettext"
411 So, what needs to replace gettext is a system that supports lexicons
412 of functions instead of lexicons of strings. An entry in a lexicon
413 from such a system should \fInot\fR look like this:
414 .PP
415 .Vb 1
416 \& "J\*(Aqai trouv\exE9 %g fichiers dans %g r\exE9pertoires"
417 .Ve
418 .PP
419 [\exE9 is e\-acute in Latin\-1. Some pod renderers would
420 scream if I used the actual character here. \-\- SB]
421 .PP
422 but instead like this, bearing in mind that this is just a first stab:
423 .PP
424 .Vb 8
425 \& sub I_found_X1_files_in_X2_directories {
426 \& my( $files, $dirs ) = @_[0,1];
427 \& $files = sprintf("%g %s", $files,
428 \& $files == 1 ? \*(Aqfichier\*(Aq : \*(Aqfichiers\*(Aq);
429 \& $dirs = sprintf("%g %s", $dirs,
430 \& $dirs == 1 ? "r\exE9pertoire" : "r\exE9pertoires");
431 \& return "J\*(Aqai trouv\exE9 $files dans $dirs.";
432 \& }
433 .Ve
434 .PP
435 Now, there's no particularly obvious way to store anything but strings
436 in a gettext lexicon; so it looks like we just have to start over and
437 make something better, from scratch. I call my shot at a
438 gettext-replacement system "Maketext", or, in CPAN terms,
439 Locale::Maketext.
440 .PP
441 When designing Maketext, I chose to plan its main features in terms of
442 "buzzword compliance". And here are the buzzwords:
443 .SS "Buzzwords: Abstraction and Encapsulation"
444 .IX Subsection "Buzzwords: Abstraction and Encapsulation"
445 The complexity of the language you're trying to output a phrase in is
446 entirely abstracted inside (and encapsulated within) the Maketext module
447 for that interface. When you call:
448 .PP
449 .Vb 2
450 \& print $lang\->maketext("You have [quant,_1,piece] of new mail.",
451 \& scalar(@messages));
452 .Ve
453 .PP
454 you don't know (and in fact can't easily find out) whether this will
455 involve lots of figuring, as in Russian (if \f(CW$lang\fR is a handle to the
456 Russian module), or relatively little, as in Chinese. That kind of
457 abstraction and encapsulation may encourage other pleasant buzzwords
458 like modularization and stratification, depending on what design
459 decisions you make.
460 .SS "Buzzword: Isomorphism"
461 .IX Subsection "Buzzword: Isomorphism"
462 "Isomorphism" means "having the same structure or form"; in discussions
463 of program design, the word takes on the special, specific meaning that
464 your implementation of a solution to a problem \fIhas the same
465 structure\fR as, say, an informal verbal description of the solution, or
466 maybe of the problem itself. Isomorphism is, all things considered,
467 a good thing \-\- it's what problem-solving (and solution-implementing)
468 should look like.
469 .PP
470 What's wrong the with gettext-using code like this...
471 .PP
472 .Vb 9
473 \& printf( $file_count == 1 ?
474 \& ( $directory_count == 1 ?
475 \& "Your query matched %g file in %g directory." :
476 \& "Your query matched %g file in %g directories." ) :
477 \& ( $directory_count == 1 ?
478 \& "Your query matched %g files in %g directory." :
479 \& "Your query matched %g files in %g directories." ),
480 \& $file_count, $directory_count,
481 \& );
482 .Ve
483 .PP
484 is first off that it's not well abstracted \-\- these ways of testing
485 for grammatical number (as in the expressions like \f(CW\*(C`foo == 1 ?
486 singular_form : plural_form\*(C'\fR) should be abstracted to each language
487 module, since how you get grammatical number is language-specific.
488 .PP
489 But second off, it's not isomorphic \-\- the "solution" (i.e., the
490 phrasebook entries) for Chinese maps from these four English phrases to
491 the one Chinese phrase that fits for all of them. In other words, the
492 informal solution would be "The way to say what you want in Chinese is
493 with the one phrase 'For your question, in Y directories you would
494 find X files'" \-\- and so the implemented solution should be,
495 isomorphically, just a straightforward way to spit out that one
496 phrase, with numerals properly interpolated. It shouldn't have to map
497 from the complexity of other languages to the simplicity of this one.
498 .SS "Buzzword: Inheritance"
499 .IX Subsection "Buzzword: Inheritance"
500 There's a great deal of reuse possible for sharing of phrases between
501 modules for related dialects, or for sharing of auxiliary functions
502 between related languages. (By "auxiliary functions", I mean
503 functions that don't produce phrase-text, but which, say, return an
504 answer to "does this number require a plural noun after it?". Such
505 auxiliary functions would be used in the internal logic of functions
506 that actually do produce phrase-text.)
507 .PP
508 In the case of sharing phrases, consider that you have an interface
509 already localized for American English (probably by having been
510 written with that as the native locale, but that's incidental).
511 Localizing it for UK English should, in practical terms, be just a
512 matter of running it past a British person with the instructions to
513 indicate what few phrases would benefit from a change in spelling or
514 possibly minor rewording. In that case, you should be able to put in
515 the UK English localization module \fIonly\fR those phrases that are
516 UK-specific, and for all the rest, \fIinherit\fR from the American
517 English module. (And I expect this same situation would apply with
518 Brazilian and Continental Portugese, possibly with some \fIvery\fR
519 closely related languages like Czech and Slovak, and possibly with the
520 slightly different "versions" of written Mandarin Chinese, as I hear exist in
521 Taiwan and mainland China.)
522 .PP
523 As to sharing of auxiliary functions, consider the problem of Russian
524 numbers from the beginning of this article; obviously, you'd want to
525 write only once the hairy code that, given a numeric value, would
526 return some specification of which case and number a given quantified
527 noun should use. But suppose that you discover, while localizing an
528 interface for, say, Ukrainian (a Slavic language related to Russian,
529 spoken by several million people, many of whom would be relieved to
530 find that your Web site's or software's interface is available in
531 their language), that the rules in Ukrainian are the same as in Russian
532 for quantification, and probably for many other grammatical functions.
533 While there may well be no phrases in common between Russian and
534 Ukrainian, you could still choose to have the Ukrainian module inherit
535 from the Russian module, just for the sake of inheriting all the
536 various grammatical methods. Or, probably better organizationally,
537 you could move those functions to a module called \f(CW\*(C`_E_Slavic\*(C'\fR or
538 something, which Russian and Ukrainian could inherit useful functions
539 from, but which would (presumably) provide no lexicon.
540 .SS "Buzzword: Concision"
541 .IX Subsection "Buzzword: Concision"
542 Okay, concision isn't a buzzword. But it should be, so I decree that
543 as a new buzzword, "concision" means that simple common things should
544 be expressible in very few lines (or maybe even just a few characters)
545 of code \-\- call it a special case of "making simple things easy and
546 hard things possible", and see also the role it played in the
547 MIDI::Simple language, discussed elsewhere in this issue [TPJ#13].
548 .PP
549 Consider our first stab at an entry in our "phrasebook of functions":
550 .PP
551 .Vb 8
552 \& sub I_found_X1_files_in_X2_directories {
553 \& my( $files, $dirs ) = @_[0,1];
554 \& $files = sprintf("%g %s", $files,
555 \& $files == 1 ? \*(Aqfichier\*(Aq : \*(Aqfichiers\*(Aq);
556 \& $dirs = sprintf("%g %s", $dirs,
557 \& $dirs == 1 ? "r\exE9pertoire" : "r\exE9pertoires");
558 \& return "J\*(Aqai trouv\exE9 $files dans $dirs.";
559 \& }
560 .Ve
561 .PP
562 You may sense that a lexicon (to use a non-committal catch-all term for a
563 collection of things you know how to say, regardless of whether they're
564 phrases or words) consisting of functions \fIexpressed\fR as above would
565 make for rather long-winded and repetitive code \-\- even if you wisely
566 rewrote this to have quantification (as we call adding a number
567 expression to a noun phrase) be a function called like:
568 .PP
569 .Vb 6
570 \& sub I_found_X1_files_in_X2_directories {
571 \& my( $files, $dirs ) = @_[0,1];
572 \& $files = quant($files, "fichier");
573 \& $dirs = quant($dirs, "r\exE9pertoire");
574 \& return "J\*(Aqai trouv\exE9 $files dans $dirs.";
575 \& }
576 .Ve
577 .PP
578 And you may also sense that you do not want to bother your translators
579 with having to write Perl code \-\- you'd much rather that they spend
580 their \fIvery costly time\fR on just translation. And this is to say
581 nothing of the near impossibility of finding a commercial translator
582 who would know even simple Perl.
583 .PP
584 In a first-hack implementation of Maketext, each language-module's
585 lexicon looked like this:
586 .PP
587 .Vb 10
588 \& %Lexicon = (
589 \& "I found %g files in %g directories"
590 \& => sub {
591 \& my( $files, $dirs ) = @_[0,1];
592 \& $files = quant($files, "fichier");
593 \& $dirs = quant($dirs, "r\exE9pertoire");
594 \& return "J\*(Aqai trouv\exE9 $files dans $dirs.";
595 \& },
596 \& ... and so on with other phrase => sub mappings ...
597 \& );
598 .Ve
599 .PP
600 but I immediately went looking for some more concise way to basically
601 denote the same phrase-function \-\- a way that would also serve to
602 concisely denote \fImost\fR phrase-functions in the lexicon for \fImost\fR
603 languages. After much time and even some actual thought, I decided on
604 this system:
605 .PP
606 * Where a value in a \f(CW%Lexicon\fR hash is a contentful string instead of
607 an anonymous sub (or, conceivably, a coderef), it would be interpreted
608 as a sort of shorthand expression of what the sub does. When accessed
609 for the first time in a session, it is parsed, turned into Perl code,
610 and then eval'd into an anonymous sub; then that sub replaces the
611 original string in that lexicon. (That way, the work of parsing and
612 evaling the shorthand form for a given phrase is done no more than
613 once per session.)
614 .PP
615 * Calls to \f(CW\*(C`maketext\*(C'\fR (as Maketext's main function is called) happen
616 thru a "language session handle", notionally very much like an IO
617 handle, in that you open one at the start of the session, and use it
618 for "sending signals" to an object in order to have it return the text
619 you want.
620 .PP
621 So, this:
622 .PP
623 .Vb 2
624 \& $lang\->maketext("You have [quant,_1,piece] of new mail.",
625 \& scalar(@messages));
626 .Ve
627 .PP
628 basically means this: look in the lexicon for \f(CW$lang\fR (which may inherit
629 from any number of other lexicons), and find the function that we
630 happen to associate with the string "You have [quant,_1,piece] of new
631 mail" (which is, and should be, a functioning "shorthand" for this
632 function in the native locale \-\- English in this case). If you find
633 such a function, call it with \f(CW$lang\fR as its first parameter (as if it
634 were a method), and then a copy of scalar(@messages) as its second,
635 and then return that value. If that function was found, but was in
636 string shorthand instead of being a fully specified function, parse it
637 and make it into a function before calling it the first time.
638 .PP
639 * The shorthand uses code in brackets to indicate method calls that
640 should be performed. A full explanation is not in order here, but a
641 few examples will suffice:
642 .PP
643 .Vb 1
644 \& "You have [quant,_1,piece] of new mail."
645 .Ve
646 .PP
647 The above code is shorthand for, and will be interpreted as,
648 this:
649 .PP
650 .Vb 8
651 \& sub {
652 \& my $handle = $_[0];
653 \& my(@params) = @_;
654 \& return join \*(Aq\*(Aq,
655 \& "You have ",
656 \& $handle\->quant($params[1], \*(Aqpiece\*(Aq),
657 \& "of new mail.";
658 \& }
659 .Ve
660 .PP
661 where "quant" is the name of a method you're using to quantify the
662 noun "piece" with the number \f(CW$params\fR[0].
663 .PP
664 A string with no brackety calls, like this:
665 .PP
666 .Vb 1
667 \& "Your search expression was malformed."
668 .Ve
669 .PP
670 is somewhat of a degenerate case, and just gets turned into:
671 .PP
672 .Vb 1
673 \& sub { return "Your search expression was malformed." }
674 .Ve
675 .PP
676 However, not everything you can write in Perl code can be written in
677 the above shorthand system \-\- not by a long shot. For example, consider
678 the Italian translator from the beginning of this article, who wanted
679 the Italian for "I didn't find any files" as a special case, instead
680 of "I found 0 files". That couldn't be specified (at least not easily
681 or simply) in our shorthand system, and it would have to be written
682 out in full, like this:
683 .PP
684 .Vb 10
685 \& sub { # pretend the English strings are in Italian
686 \& my($handle, $files, $dirs) = @_[0,1,2];
687 \& return "I didn\*(Aqt find any files" unless $files;
688 \& return join \*(Aq\*(Aq,
689 \& "I found ",
690 \& $handle\->quant($files, \*(Aqfile\*(Aq),
691 \& " in ",
692 \& $handle\->quant($dirs, \*(Aqdirectory\*(Aq),
693 \& ".";
694 \& }
695 .Ve
696 .PP
697 Next to a lexicon full of shorthand code, that sort of sticks out like a
698 sore thumb \-\- but this \fIis\fR a special case, after all; and at least
699 it's possible, if not as concise as usual.
700 .PP
701 As to how you'd implement the Russian example from the beginning of
702 the article, well, There's More Than One Way To Do It, but it could be
703 something like this (using English words for Russian, just so you know
704 what's going on):
705 .PP
706 .Vb 1
707 \& "I [quant,_1,directory,accusative] scanned."
708 .Ve
709 .PP
710 This shifts the burden of complexity off to the quant method. That
711 method's parameters are: the numeric value it's going to use to
712 quantify something; the Russian word it's going to quantify; and the
713 parameter "accusative", which you're using to mean that this
714 sentence's syntax wants a noun in the accusative case there, although
715 that quantification method may have to overrule, for grammatical
716 reasons you may recall from the beginning of this article.
717 .PP
718 Now, the Russian quant method here is responsible not only for
719 implementing the strange logic necessary for figuring out how Russian
720 number-phrases impose case and number on their noun-phrases, but also
721 for inflecting the Russian word for "directory". How that inflection
722 is to be carried out is no small issue, and among the solutions I've
723 seen, some (like variations on a simple lookup in a hash where all
724 possible forms are provided for all necessary words) are
725 straightforward but \fIcan\fR become cumbersome when you need to inflect
726 more than a few dozen words; and other solutions (like using
727 algorithms to model the inflections, storing only root forms and
728 irregularities) \fIcan\fR involve more overhead than is justifiable for
729 all but the largest lexicons.
730 .PP
731 Mercifully, this design decision becomes crucial only in the hairiest
732 of inflected languages, of which Russian is by no means the \fIworst\fR case
733 scenario, but is worse than most. Most languages have simpler
734 inflection systems; for example, in English or Swahili, there are
735 generally no more than two possible inflected forms for a given noun
736 ("error/errors"; "kosa/makosa"), and the
737 rules for producing these forms are fairly simple \-\- or at least,
738 simple rules can be formulated that work for most words, and you can
739 then treat the exceptions as just "irregular", at least relative to
740 your ad hoc rules. A simpler inflection system (simpler rules, fewer
741 forms) means that design decisions are less crucial to maintaining
742 sanity, whereas the same decisions could incur
743 overhead-versus-scalability problems in languages like Russian. It
744 may \fIalso\fR be likely that code (possibly in Perl, as with
745 Lingua::EN::Inflect, for English nouns) has already
746 been written for the language in question, whether simple or complex.
747 .PP
748 Moreover, a third possibility may even be simpler than anything
749 discussed above: "Just require that all possible (or at least
750 applicable) forms be provided in the call to the given language's quant
751 method, as in:"
752 .PP
753 .Vb 1
754 \& "I found [quant,_1,file,files]."
755 .Ve
756 .PP
757 That way, quant just has to chose which form it needs, without having
758 to look up or generate anything. While possibly not optimal for
759 Russian, this should work well for most other languages, where
760 quantification is not as complicated an operation.
761 .SS "The Devil in the Details"
762 .IX Subsection "The Devil in the Details"
763 There's plenty more to Maketext than described above \-\- for example,
764 there's the details of how language tags ("en-US", "i\-pwn", "fi",
765 etc.) or locale IDs ("en_US") interact with actual module naming
766 ("BogoQuery/Locale/en_us.pm"), and what magic can ensue; there's the
767 details of how to record (and possibly negotiate) what character
768 encoding Maketext will return text in (UTF8? Latin\-1? KOI8?). There's
769 the interesting fact that Maketext is for localization, but nowhere
770 actually has a "\f(CW\*(C`use locale;\*(C'\fR" anywhere in it. For the curious,
771 there's the somewhat frightening details of how I actually
772 implement something like data inheritance so that searches across
773 modules' \f(CW%Lexicon\fR hashes can parallel how Perl implements method
774 inheritance.
775 .PP
776 And, most importantly, there's all the practical details of how to
777 actually go about deriving from Maketext so you can use it for your
778 interfaces, and the various tools and conventions for starting out and
779 maintaining individual language modules.
780 .PP
781 That is all covered in the documentation for Locale::Maketext and the
782 modules that come with it, available in CPAN. After having read this
783 article, which covers the why's of Maketext, the documentation,
784 which covers the how's of it, should be quite straightforward.
785 .SS "The Proof in the Pudding: Localizing Web Sites"
786 .IX Subsection "The Proof in the Pudding: Localizing Web Sites"
787 Maketext and gettext have a notable difference: gettext is in C,
788 accessible thru C library calls, whereas Maketext is in Perl, and
789 really can't work without a Perl interpreter (although I suppose
790 something like it could be written for C). Accidents of history (and
791 not necessarily lucky ones) have made C++ the most common language for
792 the implementation of applications like word processors, Web browsers,
793 and even many in-house applications like custom query systems. Current
794 conditions make it somewhat unlikely that the next one of any of these
795 kinds of applications will be written in Perl, albeit clearly more for
796 reasons of custom and inertia than out of consideration of what is the
797 right tool for the job.
798 .PP
799 However, other accidents of history have made Perl a well-accepted
800 language for design of server-side programs (generally in CGI form)
801 for Web site interfaces. Localization of static pages in Web sites is
802 trivial, feasible either with simple language-negotiation features in
803 servers like Apache, or with some kind of server-side inclusions of
804 language-appropriate text into layout templates. However, I think
805 that the localization of Perl-based search systems (or other kinds of
806 dynamic content) in Web sites, be they public or access-restricted,
807 is where Maketext will see the greatest use.
808 .PP
809 I presume that it would be only the exceptional Web site that gets
810 localized for English \fIand\fR Chinese \fIand\fR Italian \fIand\fR Arabic
811 \&\fIand\fR Russian, to recall the languages from the beginning of this
812 article \-\- to say nothing of German, Spanish, French, Japanese,
813 Finnish, and Hindi, to name a few languages that benefit from large
814 numbers of programmers or Web viewers or both.
815 .PP
816 However, the ever-increasing internationalization of the Web (whether
817 measured in terms of amount of content, of numbers of content writers
818 or programmers, or of size of content audiences) makes it increasingly
819 likely that the interface to the average Web-based dynamic content
820 service will be localized for two or maybe three languages. It is my
821 hope that Maketext will make that task as simple as possible, and will
822 remove previous barriers to localization for languages dissimilar to
823 English.
824 .PP
825 .Vb 1
826 \& _\|_END_\|_
827 .Ve
828 .PP
829 Sean M. Burke ([email protected]) has a Master's in linguistics
830 from Northwestern University; he specializes in language technology.
831 Jordan Lachler ([email protected]) is a PhD student in the Department of
832 Linguistics at the University of New Mexico; he specializes in
833 morphology and pedagogy of North American native languages.
834 .SS References
835 .IX Subsection "References"
836 Alvestrand, Harald Tveit. 1995. \fIRFC 1766: Tags for the
837 Identification of Languages.\fR
838 \&\f(CW\*(C`<http://www.ietf.org/rfc/rfc1766.txt>\*(C'\fR
839 [Now see RFC 3066.]
840 .PP
841 Callon, Ross, editor. 1996. \fIRFC 1925: The Twelve
842 Networking Truths.\fR
843 \&\f(CW\*(C`<http://www.ietf.org/rfc/rfc1925.txt>\*(C'\fR
844 .PP
845 Drepper, Ulrich, Peter Miller,
846 and François Pinard. 1995\-2001. GNU
847 \&\f(CW\*(C`gettext\*(C'\fR. Available in \f(CW\*(C`<ftp://prep.ai.mit.edu/pub/gnu/>\*(C'\fR, with
848 extensive docs in the distribution tarball. [Since
849 I wrote this article in 1998, I now see that the
850 gettext docs are now trying more to come to terms with
851 plurality. Whether useful conclusions have come from it
852 is another question altogether. \-\- SMB, May 2001]
853 .PP
854 Forbes, Nevill. 1964. \fIRussian Grammar.\fR Third Edition, revised
855 by J. C. Dumbreck. Oxford University Press.