Debian Codegraph @

texmf-dist/doc/luatex/interpreter/interpreter-doc.txt

raw
1 TABLE OF CONTENTS
2 (Make a search on the tag in the right column to jump
3 to the associated section.
4 Vim users can simply type * on the tag;
5 Emacs users do that with C-s C-w (I think);
6 Other editors: I don't know!)
7 ===========================================================
8 Introduction intro_tag
9 Input files input_tag
10 Paragraphs paragraphs_tag
11 Declaring patterns patterns_tag
12 Classes classes_tag
13 Protecting input protect_tag
14 Technical stuff technical_tag
15 An example: i-doc.lua example_tag
16 The Gates in Interpreter gates_tag
17 The interpreter table interpreter_tag
18 The interpreter.core.tools table interpreter_tools_tag
19 The interpreter.core.reader table interpreter_reader_tag
20 ===========================================================
21
22
23
24 ================================== intro_tag
25 === Introduction =================
26 ==================================
27
28 Interpreter preprocesses input files before their contents is fed to TeX. It
29 is meant to write document with whatever markup one wishes to define while
30 using normal TeX macros in the background. As a simple example, suppose you
31 have a macro "\bold" to put text in boldface; then Interpreter lets you map
32 "*text*", or "<strong>text</strong>", or simply "!text", or anything else, to
33 "\bold{text}". Interpreter doesn't perform any trickery with active
34 characters; instead, it manipulates the strings representing the lines of
35 a file and search for patterns.
36
37 There are two main advantages: first, TeX documents can be typeset with
38 a completely non-TeX syntax; second, if one uses some lightweight markup
39 language, the source file is much easier to read and might even be more useful
40 than the typeset PDF file, e.g. for some technical documentation you want to
41 read directly in your text editor while writing code (powerful editors
42 generally have their own documentation in such a format, for a good reason).
43 A third advantage, not explored in this documentation, is that while feeding
44 modified lines to TeX you can also translate the original lines into, say,
45 HTML, and write them to an external file, thus creating both PDF and HTML
46 output at once.
47
48 Interpreter has been rewritten with the Gates package (actually, only
49 the Lua side) in version 1.1. That hasn't changed anything to its default
50 behavior, but now it can also be customized quite deeply, since its
51 code is a collection of small chunks with names that can be externally
52 controlled and/or augmented. See the Gates documentation for further
53 information. The last sections of this documentation describe the gates
54 in Interpreter.
55
56
57
58 ================================== input_tag
59 === Input files ==================
60 ==================================
61
62 Once Interpreter is loaded with
63
64 \input interpreter
65
66 in plain TeX or
67
68 \usepackage{interpreter}
69
70 in LaTeX, files to be processed are input as follows:
71
72 \interpretfile{<language>}{<file>}
73
74 There should exist a file "i-<language>.lua" containing the language used in
75 <file>. For instance, the source of this documentation is
76 "interpreter-doc.txt", input in the master file "interpreter-doc.tex" with
77
78 \interpretfile{doc}{interpreter-doc.txt}
79
80 and the interpretation to be used is defined in "i-doc.lua". The contents of
81 such an interpretation file is the object of the rest of this documentation.
82
83
84
85 ================================== paragraphs_tag
86 === Paragraphs ===================
87 ==================================
88
89 Interpreter doesn't process lines one by one. Instead, it gathers an entire
90 paragraph and then processes the lines. It is important because you can
91 manipulate an entire paragraph when a given pattern is detected, and modify
92 several lines according to what happens in only one. A paragraph in
93 Interpreter has nothing to do with what TeX considers a paragraph; instead, it
94 is defined by the following string.
95
96 > interpreter.paragraph [Default: blank line with spaces ignored]
97 A string to be interpreted as a paragraph boundary when Interpreter collects
98 lines before processing them. The string actually represents a pattern, so
99 magic characters are obeyed. The default is "%s*", i.e. a blank line is
100 considered a paragraph boundary, spaces notwithstanding. Of course, the end
101 of the file itself is a paragraph boundary.
102
103
104
105 ================================== patterns_tag
106 === Declaring patterns ===========
107 ==================================
108
109 Once the lines of a paragraph have been collected, Interpreter searches them
110 trying to match declared patterns, but it doesn't do so indiscriminately:
111 patterns are searched in a given order, as explained below.
112
113 Patterns are searched for in each line only, i.e. no match can occur across
114 lines. However, since you can manipulate entire paragraphs based on a match in
115 one line, the limitation easily vanishes.
116
117 > interpreter.add_pattern(<table>)
118 This is the basic function used to defined patterns. The <table> may
119 contain the following entries, along other entries Interpreter won't use
120 but which can be useful to you, especially with "call" below. The function
121 returns a table.
122
123 >> class [Default: "intepreter.default_class"]
124 The class of the pattern. See the section on classes.
125
126 >> pattern
127 The pattern to match. Lua's magic characters are in force and should be
128 escaped with "%" if necessary, unless "nomagic" is "true" (or the pattern
129 itself is the result of "interpreter.nomagic").
130
131 >> nomagic [Default: "false"]
132 A boolean deciding whether the pattern should be transformed with
133 "interpreter.nomagic".
134
135 >> replace
136 The replacement for the pattern, applied only if there is no "call" entry.
137 This may be a string, a table or a function. Interpreter simply executes
138 something similar to "string.gsub()", hence the replacement follows this
139 function's ordinary syntax. More precisely, if "replace" is a string, the
140 pattern is replaced with it; in this string, "%n" may be used to denote the
141 _n_th capture in the pattern. If "replace" is a table, the first capture or
142 the entire match (if there is no capture) is used as the key, and the
143 associated value is used as the replacement. If "replace" is a function, it
144 is called with the captures passed as arguments, or the entire match if
145 there is no capture. For instance, the following pattern will replace all
146 "*text*" with "\bold{text}":
147
148 interpreter.add_pattern{
149 pattern = "%*(.-)%*",
150 replace = [[\bold{%1}]]
151 }
152
153 >> offset [Default: 0]
154 The number of positions Interpreter should shift to the right after a match
155 has occurred. Normally, Interpreter starts searching for another occurrence
156 of the current pattern at the same position where it found the last one.
157 However, loops might easily occur: the replacement for a pattern may very
158 well contain another match for the same pattern, so Interpreter will get
159 stuck. Suppose for instance you want to replace "TeX" with "\TeX". The
160 first match will do that, but then Interpreter will start searching again
161 at the backslash, producing "\\TeX", then "\\\TeX", etc. In this case, if
162 you set "offset" to 2 in the pattern, then search will start again at the
163 "e" and no new match will occur.
164
165 >> call
166 This entry shall contain a function to be called if there is a match (if
167 this entry exists, "replace" isn't applied). It is meant to perform complex
168 tasks that aren't amenable to simple string replacement. The function will
169 be executed as follows:
170
171 function (paragraph, line, index, pattern)
172
173 "paragraph" is a table representing the current paragraph; lines are stored
174 at successive indices. The last line of this paragraph is always the
175 paragraph boundary (see "interpreter.paragraph"), unless the paragraph
176 stopped at the end of the file. The second argument, "line", is a number
177 representing the index in "paragraph" containing the line where the pattern
178 was found; "index" is the position in this line where the match occurred.
179 Finally, "pattern" is the entire table declared with
180 "interpreter.add_pattern" and containing all the entries discussed here.
181
182 The function may return zero, one, or two numbers. If it returns none, the
183 search for the next occurrence of the pattern will start again on the same
184 line (rather, on the line with the same position in the paragraph), at
185 "index". If it returns one number, the search will resume at the same line
186 but at position _n_, with _n_ the returned number. Finally, if two numbers
187 are returned, the search will resume at line _m_ at position _n_, _m_ and
188 _n_ being the returned values. Specifying which line should be examined
189 when the search resumes might be necessary if the function adds new lines
190 in the paragraph _before_ the current line, since Interpreter only keeps
191 count of line numbers.
192
193 The entire paragraph can thus be modified if necessary. For instance,
194 suppose you want to declare comments in your source file with only
195 "!Comment" in the first line, i.e. TeX should ignore a paragraph such as:
196
197 !Comment
198 This should be ignored
199 by TeX
200
201 Then the following pattern will do (where the function requires only the
202 first argument):
203
204 local function comment (paragraph)
205 for n, l in ipairs(paragraph) do
206 paragraph[n] = "%" .. l
207 end
208 end
209 interpreter.add_pattern{
210 pattern = "^!Comment",
211 call = comment
212 }
213
214 > interpreter.nomagic (string)
215 A function which reverses the usual Lua magic for patterns: ordinary magic
216 characters are normal characters here, unless they are prefixed with "%", in
217 which case they are magic again. For instance, a pattern like ".+" is
218 normally interpreted as ``one or more characters''. If passed to this
219 function, a pattern is returned meaning ``a dot followed by a plus sign''.
220 On the contrary, "%.%+" normally has the second interpretation, while with
221 "interpreter.nomagic" it has the first one. The function makes another
222 transformation: "..." is used to denote a capture "(.-)". Thus
223 "interpreter.nomagic('*...*')" returns a pattern matching any number of
224 characters surrounded by stars and capturing those characters; this would be
225 expressed in ordinary Lua magic as "%*(.-)%*".
226
227
228
229 ================================== classes_tag
230 === Classes ======================
231 ==================================
232
233 As already alluded to, the search for patterns isn't done at random. Instead,
234 patterns are organized in classes, which are applied one after the other. More
235 precisely, the process is as follows: Interpreter searches the entire
236 paragraph for the first pattern in class~1, then for the second pattern in the
237 same class, then for the third, etc., then when there is no pattern left in
238 class~1 it does the same with class~2, up to class~_n_, where _n_ is the
239 highest class number such that there exists a class _n - 1_ (in other words,
240 classes should be numbered consecutively). Finally, the same goes for the
241 patterns in class~0 (which always exists, even if it contains no pattern).
242
243 Inside a class, patterns are ordered by length from long to short, or
244 alphabetically if two patterns have the same length. This means that if you
245 use e.g. "/text/" for italics and "//text//" for bold, you don't need to put
246 the second pattern in a class before the first to avoid "//text//" being
247 interpreted as two empty arguments in italics surrounding a text in roman.
248 Since the way the bold-pattern will be declared, e.g. "//(.-)//", is probably
249 longer than for the italic-pattern, e.g. "/(.-)/", it will always match first.
250
251 That said, the sorting isn't very clever and simply relies on the number of
252 symbols, no matter what they mean; in the patterns above, the parentheses
253 denote a capture but they still count in the pattern's length as understood by
254 Interpreter. Alternatively, while ".*" denotes ``zero or more character'' and
255 "%+" means ``a plus sign'' ("+" being magic, you have to escape it to refer to
256 it), in Interpreter's eye the two patterns have the same length: two. Finally,
257 one should be aware that patterns declared with a "nomagic" entry set to
258 "true" are sorted after they've been transformed (so that their real length
259 might not be obvious). So classes are needed when patterns need a proper
260 ordering no matter their lengths. For instance, some patterns should always be
261 declared first, as they protect input from Interpreter (see next section),
262 while others might need to be declared last, as they rely on what previous
263 patterns might have done. Besides, classes are metatables for the patterns
264 they contain.
265
266 > interpreter.default_class [Default: 1]
267 All patterns belong to a class, even though you may omit the "class" entry
268 when declaring one. In this case, the pattern is assigned to the class
269 denoted by this number.
270
271 > interpreter.set_class(number, table)
272 Defines class "number" as "table". Classes don't need to be defined
273 beforehand for patterns to be added to them (rather, Interpreter defines
274 them implicitly when needed). However, classes are also metatables for the
275 patterns, so that if there lacks an entry in a pattern's table, the class's
276 entry is used if it exists. The function returns a table.
277
278
279
280 ================================== protect_tag
281 === Protecting input =============
282 ==================================
283
284 Sometimes you want Interpreter to refrain from interpreting; that is most
285 useful for verbatim code, for instance. There are various ways to do that.
286
287 > interpreter.active [Default: true]
288 A boolean switching Interpreter on and off. Beware, the switching applies
289 only starting at the next paragraph.
290
291 > interpreter.protect([line])
292 A function protecting all or part of the current paragraph. If "line" is
293 given, it should be a number _n_, and line _n_ in the current paragraph will
294 be protected; without "line", the entire paragraph is protected. Protecting
295 means that the patterns not yet searched for will be ignored. For instance,
296 if you want material to be read verbatim when surrounded with "<code>" and
297 "</code>", you can declare a pattern as follows:
298
299 local function verbatim (buffer)
300 buffer[1] = "\\verbatim"
301 buffer[#buffer - 1] = "\\endverbatim"
302 intepreter.protect()
303 end
304 interpreter.add_pattern{
305 pattern = "^%s*<code>%*s$",
306 call = verbatim,
307 class = 1
308 }
309
310 This code is extremely simplified : it assumes that "<code>" and "</code>"
311 starts and ends the paragraph and that "</code>" isn't the last line of the
312 file (otherwise it'd also be the last line in the paragraph, whereas here
313 the last one is the paragraph boundary). An important point is that the
314 pattern belongs to the first class, so it is called before all other
315 patterns (provided there is no shorter pattern in class~1) and prevents them
316 from doing anything, since the entire paragraph is protected. (Typesetting
317 the material as verbatim material obviously depends on the "\verbatim"
318 macro, not on Interpreter.)
319
320 > interpreter.escape
321 A character which prevents patterns from being replaced if immediately
322 preceded by it. As an example, if "interpreter.escape = '_'", and "*text*"
323 denotes italic, then "*text*" will produce _text_ while "_*text*" will
324 produce *text*. Once a paragraph has been processed, Interpreter removes all
325 escape characters. Only one character can be an escape character.
326
327 > interpreter.protector(left[, right]) ["right" defaults to "left"]
328 Defines two characters to protect what they surround. In other words,
329 Interpreter replaces patterns only if the match isn't found between "left"
330 and "right". Unlike the escape character, you can define as many protectors
331 as you wish; and unlike the escape character again, Interpreter _doesn't_
332 remove them once the paragraph has been processed, so you must take care of
333 them. For instance:
334
335 intepreter.protector('"')
336 interpreter.add_pattern{
337 pattern = '"(.-)"',
338 replace = '\\verb`%1`',
339 class = 0
340 }
341
342 Anything between double quotes will be left untouched; then, when the
343 paragraph has been processed for all other classes, a pattern in class~0
344 calls the "\verb" command to take care of the argument. Note that the
345 protectors should enclose what they protect without coinciding with it; this
346 is not the case here, which is why the pattern is applied.
347
348 > interpreter.direct [Default: two percent signs then "I" and at least one space]
349 A string, actually a pattern, signalling that the line which it begins
350 should be processed as Lua code. The default is "%%%%I%s+", i.e. "%%I"
351 followed by at least one space. The pattern shouldn't declare itself as
352 attached to the beginning of the line (as in "^%%%%I%s+") because they will
353 be matched at the beginning of the line only anyway. The line is processed
354 with the "loadstring" function, and then turned into an empty line. For
355 instance:
356
357 %%I interpreter.active = false
358 This won't be interpreted...
359 %%I interpreter.active = true
360
361 As this example shows, lines flagged with "interpreter.direct" don't obey
362 "interpreter.active" and are always processed as described above.
363
364
365
366 ================================== technical_tag
367 === Technical stuff ==============
368 ==================================
369
370 You don't have to bother with this section if you don't mind how Interpreter
371 does its job; actually you won't learn much anyway.
372
373 > interpreter.reset()
374 A function which resets everything to default and deletes classes. It is used
375 when calling "\interpretefile" so that new interpretetions start from zero.
376
377 > interpreter.register(function)
378 A function called to put Interpreter's main function into the
379 "post_linebreak_filter" callback; you can redefine it at will. If it is
380 undefined, "callback.register()" is used, unless "luatexbase.add_to_callback()"
381 is detected. (The detection takes place at the first call to
382 "\interpretfile", so there is no need to load Interpreter after
383 "luatexbase".)
384
385 > interpreter.unregister(function)
386 A function called to remove Interpreter's main function from the
387 "post_linebreak_filter" callback. It works similarly to the previous one.
388
389
390
391 ================================== example_tag
392 === An example: i-doc.lua ========
393 ==================================
394
395 \interpretfile{doc}{i-doc.lua}
396
397
398
399 ================================== gates_tag
400 === The Gates in Interpreter =====
401 ==================================
402
403 Interpreter is written with the Gates package (only the Lua side,
404 actually). It means that it can be hacked down to the core. Here I'll
405 simply list the gates involved; you should read the Gates
406 documentation to learn how to use them.
407
408 There are three gates families: "interpreter", associated with the main
409 "interpreter" table, contains the user interface; "interpreter_tools",
410 associated with "interpreter.core.tools" table, contains internal
411 functions; finally "interpreter_reader", associated with the
412 "interpreter.core.reader" table, contains the main functions used to
413 read the file.
414
415 Whenever I mention a conditional or a loop, I mean the local conditionals
416 and loops, relative to the l-gate where the gate appears. Also, the
417 syntax indicates the arguments a gate uses, not all the arguments that
418 are passed to it (which are simply what the previous gate has returned).
419
420 As an example of customizing Interpreter with Gates, you could very
421 well add a bit of code which does something to all lines. Inserting a
422 small gate, say "everyline", after "check_direct" in "aggregate_lines"
423 below would do the trick, e.g.:
424
425
426 function interpreter.core.reader.everyline (file, line)
427 line = dosomething(line)
428 return file, line
429 end
430 interpreter.core.reader.add(
431 "everyline", "aggregate_lines", "after check_direct")
432 interpreter.core.reader.conditional(
433 "everyline", "aggregate_lines", function (f, l) return l end)
434
435 (Note that it is important to check that the line really exist,
436 because one might have hit the end of the file; hence the conditional,
437 as with others gates in "aggregate_lines").
438
439
440
441
442 ================================== interpreter_tag
443 === The "interpreter" table ========
444 ==================================
445
446 All the user functions in "interpreter" are simple m-gates, so they can
447 be treated as ordinary functions, except "interpreter.add_pattern",
448 which is an l-gate containing, built as:
449
450 add_pattern
451 . ensure_class
452 . apply_nomagic
453 . insert-pattern
454 . . do_insert
455 . . sort_class
456
457 > ensure_class (<pattern>) [m-gate]
458 Creates the class of <pattern> if necessary, and set it as the metatable
459 for <pattern>. Classes themselves are kept in the "interpreter.core.classes"
460 table. The gate return <pattern> and the class number.
461
462 > apply_nomagic (<pattern>, <class>) [m-gate]
463 Transforms the "pattern" entry in <pattern> with "intepreter.nomagic";
464 tied to a conditional that returns <pattern>'s "nomagic" entry (so
465 the gate is executed only if "nomagic" is true); autoreturns both
466 arguments.
467
468 > insert_pattern (<pattern>, <class>) [l-gate]
469 An autoreturning l-gate containing the following two gates.
470
471 > do_insert (<pattern>, <class>) [m-gate]
472 Adds <pattern> to <class> (i.e. "interpreter.core.classes[<class>]").
473
474 > sort_class (<pattern>, <class>) [m-gate]
475 Sorts <class> with function "interpreter.core.tools.sort". This gate
476 can be skipped to apply the patterns in the order in which they were
477 declared.
478
479
480
481 ============================================= interpreter_tools_tag
482 === The interpreter.core.tools table ========
483 =============================================
484
485 All the functions in the "interpreter.core.tools" all are simple m-gates.
486
487 > sort (<patt1>, <patt2>) [m-gate]
488 Returns true if the pattern in "patt1" is longer than the one in
489 "patt2", or if they have the same length and the first ranks before
490 the second with respect to alphabetical order. The gate is used in
491 the "interpreter.sort_class" m-gate.
492
493 > xsub (<string>, <index>, <pattern>, <replacement>) [m-gate]
494 Returns <string> with <pattern> replaced with <replacement>, but only
495 once, and only after <index>.
496
497 > protector (<string>, <index>) [m-gate]
498 Checks whether <index> in <string> isn't between characters declared
499 with "interpreter.protector". If that is the case, the function returns
500 "nil" and the index of the second protector. Otherwise, it returns
501 <index>.
502
503 > get_index (<string>, <pattern>, <index>) [m-gate]
504 Checks whether <pattern> occurs in <string>, starting at <index>. If
505 it does, but if <index>-1 is "interpreter.escape", calls itself with
506 <index>+1. Otherwise, calls "interpreter.core.tools.protector" to
507 check whether <index> is in a protected part of the string. If so,
508 calls itself with <right>+1 instead of <index>, where <right> is the
509 second return value of of "interpreter.core.tools.protector", i.e.
510 it searches again after the right protector. If <index> is found,
511 end of story, returns <index>, otherwise returns nothing.
512
513 The "interpreter.core.tools" table also contains
514 "magic_characters", a table with an entry for each magic character in Lua except
515 `"."' and `"%"'; the values to those entries are the same characters
516 prefixed with `"%"'. The table is used by "interpreter.nomagic" to spot
517 and replace magic characters, with the dot and the percent sign dealt
518 with independantly.
519
520
521
522 ============================================= interpreter_reader_tag
523 === The interpreter.core.reader table =======
524 =============================================
525
526 Interpreter works by hooking in the "open_read_file" callback; the
527 function registered there is the "interpreter.core.reader.input"
528 l-gate, built as follows:
529
530 input
531 . unregister
532 . . set_unregister
533 . . use_unregister
534 . open_file
535 . set_reader
536
537 > unregister (<filename>) [l-gate]
538 Contains the following two m-gates; <filename> is received from
539 "input", which itself receives it from the callback, i.e. that's the
540 file that's being input (the second argument to "\interpretfile").
541 It is also automatically returned.
542
543 > set_unregister () [m-gate]
544 Sets the function to remove "input" from the callback, namely
545 "interpreter.unregister"; the gate is called only if gate
546 "interpreter.unregister" doesn't already exits. If "luatexbase" is
547 detected, the functions there are used; otherwise, "callback.register"
548 is used with "nil" as the second argument.
549
550 > use_unregister () [m-gate]
551 Calls "interpreter.unregister()". (You don't want the next input file
552 to be processed with Interpreter by default, that's why you remove
553 the callback function; not that the current one is nonetheless
554 processed with the current file, of course.)
555
556 > open_file (<filename>) [m-gate]
557 Returns "io.open(<filename>)".
558
559 > set_reader (<file>) [m-gate]
560 Returns a table with a "reader" entry containing a function whose
561 definition is
562
563 function ()
564 return interpreter.core.reader.read_file(f)
565 end
566
567 That's the convention for the "open_read_file" callback: it should
568 return such a table, and the function will be called each time a
569 line is required from the input file.
570
571 So most of the work is done by "interpreter.core.reader.read_file",
572 which is why it is so heavy; it receives a file handle:
573
574 read_file
575 . make_paragraph
576 . . aggregate_lines
577 . . . read_line
578 . . . check_direct
579 . . . insert_line
580 . . apply_classes
581 . . . pass_class
582 . . . . pass_pattern
583 . . . . . process_lines
584 . . . . . . switch
585 . . . . . . call
586 . . . . . . replace
587 . . . . . . protect
588 . . . unprotect
589 . . . . undo_protected
590 . . . . unprotect_lines
591 . . . remove_escape
592 . return_line
593
594 > make_paragraph (<file>) [l-gate]
595 The big l-gate that contains everything that follows, barring
596 "return_line". It is called if and only if the "interpreter.core.lines"
597 table is empty; that table is where lines of a paragraph are stored,
598 and it is emptied by "return_line".
599
600 > aggregate_lines (<file>, <line>) [l-gate]
601 The main l-gate that reads line from <file> and stores them in
602 "interpreter.core.lines". It loops until <line> is "nil" or equivalent
603 to "interpreter.paragraph". (Of course, <line> is "nil" on the first
604 iteration, but the "loopuntil" conditional is evaluated after that
605 first iteration, during which the last subgate "insert_line" will
606 probably returns a line.)
607
608 > read_line (<file>) [m-gate]
609 Reads the next line from <file>, and returns <file> and that line.
610
611 > check_direct (<file>, <line>) [m-gate]
612 If <line> begins with "interpreter.direct", removes it and use
613 "loadstring" on the resulting string. Returns <file> and <line>, the
614 latter set to an empty string is the previous operation applied. The
615 gate depends on a conditional: <line> should be non-"nil" (of
616 course), and "interpreter.direct" should be defined.
617
618 > insert_line (<file>, <line>) [m-gate]
619 Adds <line> to "interpreter.core.lines". Automatically returns the
620 two arguments (and if <line> isn't "nil" or equivalent to
621 "interpreter.paragraph", it will be executed again).
622
623 > apply_classes () [l-gate]
624 The l-gate that applies transformations to the lines, once the paragraph
625 has been gathered, with the gates that follow. For each class, it
626 will apply each pattern on each line. It depends on a conditional:
627 "interpreter.core.lines" shouldn't be empty, and "interpreter.active"
628 should be true.
629
630 > pass_class () [l-gate]
631 This gate iterates on all classes in "interpreter.core.classes" and
632 then on class 0. On each iteration it checks beforehand whether the
633 paragraph is protected, i.e. "interpreter.core.reader.protected" isn't
634 a boolean (see "unprotecte" below). On each iteration, the class
635 number and the class itself are returned. (This behavior is
636 implemented with a Gates iterator.)
637
638 > pass_pattern (<ignored>, <class>) [l-gate]
639 Same as "pass_class", except it iterates on the patterns in "class":
640 it is executed as long as "interpreter.core.reader.protected" and
641 returns the patter number and the pattern itself. (The <ignored>
642 argument isn't used; it is for the "pass_class" iterator; the same
643 holds for the following gates.)
644
645 > process_lines (<ignored>, <pattern>) [l-gate]
646 Again, this calls an iterator. It browses each line in "interpreter.core.lines"
647 and returns the line's number (provided it is valid, i.e. not a table,
648 see "protect" below), <pattern> and the current index in that line.
649 To keep track of the current line and index, two internal numbers are
650 used: "interpreter.core.reader.current_line" and
651 "interpreter.core.reader.current_index".
652
653 > switch (<line>, <pattern>, <index>) [m-gate]
654 If <pattern> has a "call" entry, it sets the "call" gate below to
655 "ajar"; otherwise, if <pattern> has a "replace" entry, it sets the
656 "replace" gate to "ajar".
657
658 > call (<line>, <pattern>, <index>) [m-gate]
659 This gate is closed by default and set to "ajar" by "switch" above.
660 If, starting at <index>, the <pattern>'s "pattern" entry can be
661 found in <line> with "interpreter.core.tools.get_index" (which makes
662 sure that protectors are obeyed and returns <newindex>, where the
663 pattern is found if it occurs), the <pattern>'s "call" entry is
664 applied as
665
666 <pattern>.call(interpreter.core.lines,
667 <line>, <newindex>, <pattern>)
668
669 This may returned zero, one or two values. If nothing is returned,
670 "interpreter.core.reader.current_index" is set to 0, which makes
671 the "process_lines" iterator consider the next line. If one value is
672 returned, it is the new current index and "process_lines" will not
673 update the line number. If two values are returned, the first is the
674 new current line number and the second the new current index.
675
676 > replace (<line>, <pattern>, <index>) [m-gate]
677 This gate is closed by default and set to "ajar" by "switch" above.
678 This tries to find the <pattern>'s "pattern" like "call" above, and if
679 it is found, it applies "interpreter.core.tools.xsub" as:
680
681 xsub (interpreter.core.lines[<line>],
682 <newindex>, <pattern>.pattern, <pattern>.replace)
683
684 where <newindex> is defined as in "call". The return value of "xsub"
685 is assigned to "interpreter.core.lines[<line>]", and the current
686 index is set to <index> plus the <pattern>'s "offset" if any. If the
687 pattern wasn't found, the current index is set to 0, which makes
688 "process_lines" turn to the next line as explained in "call".
689
690 > protect () [m-gate]
691 The "interpreter.protect()" function can either protect the whole
692 paragraph (when no argument is passed) or a single line (when a number
693 is passed). In the first case, "interpreter.core.tools.protected"
694 takes the value "true", which is checked in various gates above. In
695 the second case, "interpreter.core.tools.protected" is a table with
696 each index indicating a line to be protected. This gate implements
697 the protection in that case: it iterates on all entries in the table
698 with "pairs" and protects the line with the same index in
699 "interpreter.core.lines" by transforming it into a table (with a
700 single entry, the string representing the original line); the type
701 of the line is checked in the "process_lines" iterator above. The
702 gate's iterator doesn't take arguments, but the function itself is
703 defined as taking a number (the line).
704
705 > unprotect () [l-gate]
706 Now all the patterns in all the classes have been applied to the
707 entire paragraph, and protection must be removed. This l-gate
708 contains the following two gates.
709
710 > undo_protected () [m-gate]
711 Simply sets "interpreter.core.tools.protected" to "nil" so it is
712 ready for the next paragraph.
713
714 > unprotect_lines () [m-gate]
715 Restores all the lines in "interpreter.core.lines" as simple strings.
716 The gate uses an iterator that simply runs "ipairs" on "interpreter.core.lines",
717 so the function's definition actually takes the line's number and
718 the line itself as arguments.
719
720 > remove_escape () [m-gate]
721 If "interpreter.escape" is defined, removes all its occurrences in
722 each line of the paragraph.
723
724 This is the end of the big "make_paragraph" l-gate. It won't be called
725 again until the paragraph has been fully passed to TeX, i.e. when
726 "interpreter.core.lines" is empty.
727
728 > return_line () [m-gate]
729 Pops the first line from "interpreter.core.lines" and returns it.
730 Since this is the very last subgate of "read_file", the line is
731 passed to TeX.
732