| 1 |
TABLE OF CONTENTS |
| 2 |
(Make a search on the tag in the right column to jump |
| 3 |
to the associated section. |
| 4 |
Vim users can simply type * on the tag; |
| 5 |
Emacs users do that with C-s C-w (I think); |
| 6 |
Other editors: I don't know!) |
| 7 |
=========================================================== |
| 8 |
Introduction intro_tag |
| 9 |
Input files input_tag |
| 10 |
Paragraphs paragraphs_tag |
| 11 |
Declaring patterns patterns_tag |
| 12 |
Classes classes_tag |
| 13 |
Protecting input protect_tag |
| 14 |
Technical stuff technical_tag |
| 15 |
An example: i-doc.lua example_tag |
| 16 |
The Gates in Interpreter gates_tag |
| 17 |
The interpreter table interpreter_tag |
| 18 |
The interpreter.core.tools table interpreter_tools_tag |
| 19 |
The interpreter.core.reader table interpreter_reader_tag |
| 20 |
=========================================================== |
| 21 |
|
| 22 |
|
| 23 |
|
| 24 |
================================== intro_tag |
| 25 |
=== Introduction ================= |
| 26 |
================================== |
| 27 |
|
| 28 |
Interpreter preprocesses input files before their contents is fed to TeX. It |
| 29 |
is meant to write document with whatever markup one wishes to define while |
| 30 |
using normal TeX macros in the background. As a simple example, suppose you |
| 31 |
have a macro "\bold" to put text in boldface; then Interpreter lets you map |
| 32 |
"*text*", or "<strong>text</strong>", or simply "!text", or anything else, to |
| 33 |
"\bold{text}". Interpreter doesn't perform any trickery with active |
| 34 |
characters; instead, it manipulates the strings representing the lines of |
| 35 |
a file and search for patterns. |
| 36 |
|
| 37 |
There are two main advantages: first, TeX documents can be typeset with |
| 38 |
a completely non-TeX syntax; second, if one uses some lightweight markup |
| 39 |
language, the source file is much easier to read and might even be more useful |
| 40 |
than the typeset PDF file, e.g. for some technical documentation you want to |
| 41 |
read directly in your text editor while writing code (powerful editors |
| 42 |
generally have their own documentation in such a format, for a good reason). |
| 43 |
A third advantage, not explored in this documentation, is that while feeding |
| 44 |
modified lines to TeX you can also translate the original lines into, say, |
| 45 |
HTML, and write them to an external file, thus creating both PDF and HTML |
| 46 |
output at once. |
| 47 |
|
| 48 |
Interpreter has been rewritten with the Gates package (actually, only |
| 49 |
the Lua side) in version 1.1. That hasn't changed anything to its default |
| 50 |
behavior, but now it can also be customized quite deeply, since its |
| 51 |
code is a collection of small chunks with names that can be externally |
| 52 |
controlled and/or augmented. See the Gates documentation for further |
| 53 |
information. The last sections of this documentation describe the gates |
| 54 |
in Interpreter. |
| 55 |
|
| 56 |
|
| 57 |
|
| 58 |
================================== input_tag |
| 59 |
=== Input files ================== |
| 60 |
================================== |
| 61 |
|
| 62 |
Once Interpreter is loaded with |
| 63 |
|
| 64 |
\input interpreter |
| 65 |
|
| 66 |
in plain TeX or |
| 67 |
|
| 68 |
\usepackage{interpreter} |
| 69 |
|
| 70 |
in LaTeX, files to be processed are input as follows: |
| 71 |
|
| 72 |
\interpretfile{<language>}{<file>} |
| 73 |
|
| 74 |
There should exist a file "i-<language>.lua" containing the language used in |
| 75 |
<file>. For instance, the source of this documentation is |
| 76 |
"interpreter-doc.txt", input in the master file "interpreter-doc.tex" with |
| 77 |
|
| 78 |
\interpretfile{doc}{interpreter-doc.txt} |
| 79 |
|
| 80 |
and the interpretation to be used is defined in "i-doc.lua". The contents of |
| 81 |
such an interpretation file is the object of the rest of this documentation. |
| 82 |
|
| 83 |
|
| 84 |
|
| 85 |
================================== paragraphs_tag |
| 86 |
=== Paragraphs =================== |
| 87 |
================================== |
| 88 |
|
| 89 |
Interpreter doesn't process lines one by one. Instead, it gathers an entire |
| 90 |
paragraph and then processes the lines. It is important because you can |
| 91 |
manipulate an entire paragraph when a given pattern is detected, and modify |
| 92 |
several lines according to what happens in only one. A paragraph in |
| 93 |
Interpreter has nothing to do with what TeX considers a paragraph; instead, it |
| 94 |
is defined by the following string. |
| 95 |
|
| 96 |
> interpreter.paragraph [Default: blank line with spaces ignored] |
| 97 |
A string to be interpreted as a paragraph boundary when Interpreter collects |
| 98 |
lines before processing them. The string actually represents a pattern, so |
| 99 |
magic characters are obeyed. The default is "%s*", i.e. a blank line is |
| 100 |
considered a paragraph boundary, spaces notwithstanding. Of course, the end |
| 101 |
of the file itself is a paragraph boundary. |
| 102 |
|
| 103 |
|
| 104 |
|
| 105 |
================================== patterns_tag |
| 106 |
=== Declaring patterns =========== |
| 107 |
================================== |
| 108 |
|
| 109 |
Once the lines of a paragraph have been collected, Interpreter searches them |
| 110 |
trying to match declared patterns, but it doesn't do so indiscriminately: |
| 111 |
patterns are searched in a given order, as explained below. |
| 112 |
|
| 113 |
Patterns are searched for in each line only, i.e. no match can occur across |
| 114 |
lines. However, since you can manipulate entire paragraphs based on a match in |
| 115 |
one line, the limitation easily vanishes. |
| 116 |
|
| 117 |
> interpreter.add_pattern(<table>) |
| 118 |
This is the basic function used to defined patterns. The <table> may |
| 119 |
contain the following entries, along other entries Interpreter won't use |
| 120 |
but which can be useful to you, especially with "call" below. The function |
| 121 |
returns a table. |
| 122 |
|
| 123 |
>> class [Default: "intepreter.default_class"] |
| 124 |
The class of the pattern. See the section on classes. |
| 125 |
|
| 126 |
>> pattern |
| 127 |
The pattern to match. Lua's magic characters are in force and should be |
| 128 |
escaped with "%" if necessary, unless "nomagic" is "true" (or the pattern |
| 129 |
itself is the result of "interpreter.nomagic"). |
| 130 |
|
| 131 |
>> nomagic [Default: "false"] |
| 132 |
A boolean deciding whether the pattern should be transformed with |
| 133 |
"interpreter.nomagic". |
| 134 |
|
| 135 |
>> replace |
| 136 |
The replacement for the pattern, applied only if there is no "call" entry. |
| 137 |
This may be a string, a table or a function. Interpreter simply executes |
| 138 |
something similar to "string.gsub()", hence the replacement follows this |
| 139 |
function's ordinary syntax. More precisely, if "replace" is a string, the |
| 140 |
pattern is replaced with it; in this string, "%n" may be used to denote the |
| 141 |
_n_th capture in the pattern. If "replace" is a table, the first capture or |
| 142 |
the entire match (if there is no capture) is used as the key, and the |
| 143 |
associated value is used as the replacement. If "replace" is a function, it |
| 144 |
is called with the captures passed as arguments, or the entire match if |
| 145 |
there is no capture. For instance, the following pattern will replace all |
| 146 |
"*text*" with "\bold{text}": |
| 147 |
|
| 148 |
interpreter.add_pattern{ |
| 149 |
pattern = "%*(.-)%*", |
| 150 |
replace = [[\bold{%1}]] |
| 151 |
} |
| 152 |
|
| 153 |
>> offset [Default: 0] |
| 154 |
The number of positions Interpreter should shift to the right after a match |
| 155 |
has occurred. Normally, Interpreter starts searching for another occurrence |
| 156 |
of the current pattern at the same position where it found the last one. |
| 157 |
However, loops might easily occur: the replacement for a pattern may very |
| 158 |
well contain another match for the same pattern, so Interpreter will get |
| 159 |
stuck. Suppose for instance you want to replace "TeX" with "\TeX". The |
| 160 |
first match will do that, but then Interpreter will start searching again |
| 161 |
at the backslash, producing "\\TeX", then "\\\TeX", etc. In this case, if |
| 162 |
you set "offset" to 2 in the pattern, then search will start again at the |
| 163 |
"e" and no new match will occur. |
| 164 |
|
| 165 |
>> call |
| 166 |
This entry shall contain a function to be called if there is a match (if |
| 167 |
this entry exists, "replace" isn't applied). It is meant to perform complex |
| 168 |
tasks that aren't amenable to simple string replacement. The function will |
| 169 |
be executed as follows: |
| 170 |
|
| 171 |
function (paragraph, line, index, pattern) |
| 172 |
|
| 173 |
"paragraph" is a table representing the current paragraph; lines are stored |
| 174 |
at successive indices. The last line of this paragraph is always the |
| 175 |
paragraph boundary (see "interpreter.paragraph"), unless the paragraph |
| 176 |
stopped at the end of the file. The second argument, "line", is a number |
| 177 |
representing the index in "paragraph" containing the line where the pattern |
| 178 |
was found; "index" is the position in this line where the match occurred. |
| 179 |
Finally, "pattern" is the entire table declared with |
| 180 |
"interpreter.add_pattern" and containing all the entries discussed here. |
| 181 |
|
| 182 |
The function may return zero, one, or two numbers. If it returns none, the |
| 183 |
search for the next occurrence of the pattern will start again on the same |
| 184 |
line (rather, on the line with the same position in the paragraph), at |
| 185 |
"index". If it returns one number, the search will resume at the same line |
| 186 |
but at position _n_, with _n_ the returned number. Finally, if two numbers |
| 187 |
are returned, the search will resume at line _m_ at position _n_, _m_ and |
| 188 |
_n_ being the returned values. Specifying which line should be examined |
| 189 |
when the search resumes might be necessary if the function adds new lines |
| 190 |
in the paragraph _before_ the current line, since Interpreter only keeps |
| 191 |
count of line numbers. |
| 192 |
|
| 193 |
The entire paragraph can thus be modified if necessary. For instance, |
| 194 |
suppose you want to declare comments in your source file with only |
| 195 |
"!Comment" in the first line, i.e. TeX should ignore a paragraph such as: |
| 196 |
|
| 197 |
!Comment |
| 198 |
This should be ignored |
| 199 |
by TeX |
| 200 |
|
| 201 |
Then the following pattern will do (where the function requires only the |
| 202 |
first argument): |
| 203 |
|
| 204 |
local function comment (paragraph) |
| 205 |
for n, l in ipairs(paragraph) do |
| 206 |
paragraph[n] = "%" .. l |
| 207 |
end |
| 208 |
end |
| 209 |
interpreter.add_pattern{ |
| 210 |
pattern = "^!Comment", |
| 211 |
call = comment |
| 212 |
} |
| 213 |
|
| 214 |
> interpreter.nomagic (string) |
| 215 |
A function which reverses the usual Lua magic for patterns: ordinary magic |
| 216 |
characters are normal characters here, unless they are prefixed with "%", in |
| 217 |
which case they are magic again. For instance, a pattern like ".+" is |
| 218 |
normally interpreted as ``one or more characters''. If passed to this |
| 219 |
function, a pattern is returned meaning ``a dot followed by a plus sign''. |
| 220 |
On the contrary, "%.%+" normally has the second interpretation, while with |
| 221 |
"interpreter.nomagic" it has the first one. The function makes another |
| 222 |
transformation: "..." is used to denote a capture "(.-)". Thus |
| 223 |
"interpreter.nomagic('*...*')" returns a pattern matching any number of |
| 224 |
characters surrounded by stars and capturing those characters; this would be |
| 225 |
expressed in ordinary Lua magic as "%*(.-)%*". |
| 226 |
|
| 227 |
|
| 228 |
|
| 229 |
================================== classes_tag |
| 230 |
=== Classes ====================== |
| 231 |
================================== |
| 232 |
|
| 233 |
As already alluded to, the search for patterns isn't done at random. Instead, |
| 234 |
patterns are organized in classes, which are applied one after the other. More |
| 235 |
precisely, the process is as follows: Interpreter searches the entire |
| 236 |
paragraph for the first pattern in class~1, then for the second pattern in the |
| 237 |
same class, then for the third, etc., then when there is no pattern left in |
| 238 |
class~1 it does the same with class~2, up to class~_n_, where _n_ is the |
| 239 |
highest class number such that there exists a class _n - 1_ (in other words, |
| 240 |
classes should be numbered consecutively). Finally, the same goes for the |
| 241 |
patterns in class~0 (which always exists, even if it contains no pattern). |
| 242 |
|
| 243 |
Inside a class, patterns are ordered by length from long to short, or |
| 244 |
alphabetically if two patterns have the same length. This means that if you |
| 245 |
use e.g. "/text/" for italics and "//text//" for bold, you don't need to put |
| 246 |
the second pattern in a class before the first to avoid "//text//" being |
| 247 |
interpreted as two empty arguments in italics surrounding a text in roman. |
| 248 |
Since the way the bold-pattern will be declared, e.g. "//(.-)//", is probably |
| 249 |
longer than for the italic-pattern, e.g. "/(.-)/", it will always match first. |
| 250 |
|
| 251 |
That said, the sorting isn't very clever and simply relies on the number of |
| 252 |
symbols, no matter what they mean; in the patterns above, the parentheses |
| 253 |
denote a capture but they still count in the pattern's length as understood by |
| 254 |
Interpreter. Alternatively, while ".*" denotes ``zero or more character'' and |
| 255 |
"%+" means ``a plus sign'' ("+" being magic, you have to escape it to refer to |
| 256 |
it), in Interpreter's eye the two patterns have the same length: two. Finally, |
| 257 |
one should be aware that patterns declared with a "nomagic" entry set to |
| 258 |
"true" are sorted after they've been transformed (so that their real length |
| 259 |
might not be obvious). So classes are needed when patterns need a proper |
| 260 |
ordering no matter their lengths. For instance, some patterns should always be |
| 261 |
declared first, as they protect input from Interpreter (see next section), |
| 262 |
while others might need to be declared last, as they rely on what previous |
| 263 |
patterns might have done. Besides, classes are metatables for the patterns |
| 264 |
they contain. |
| 265 |
|
| 266 |
> interpreter.default_class [Default: 1] |
| 267 |
All patterns belong to a class, even though you may omit the "class" entry |
| 268 |
when declaring one. In this case, the pattern is assigned to the class |
| 269 |
denoted by this number. |
| 270 |
|
| 271 |
> interpreter.set_class(number, table) |
| 272 |
Defines class "number" as "table". Classes don't need to be defined |
| 273 |
beforehand for patterns to be added to them (rather, Interpreter defines |
| 274 |
them implicitly when needed). However, classes are also metatables for the |
| 275 |
patterns, so that if there lacks an entry in a pattern's table, the class's |
| 276 |
entry is used if it exists. The function returns a table. |
| 277 |
|
| 278 |
|
| 279 |
|
| 280 |
================================== protect_tag |
| 281 |
=== Protecting input ============= |
| 282 |
================================== |
| 283 |
|
| 284 |
Sometimes you want Interpreter to refrain from interpreting; that is most |
| 285 |
useful for verbatim code, for instance. There are various ways to do that. |
| 286 |
|
| 287 |
> interpreter.active [Default: true] |
| 288 |
A boolean switching Interpreter on and off. Beware, the switching applies |
| 289 |
only starting at the next paragraph. |
| 290 |
|
| 291 |
> interpreter.protect([line]) |
| 292 |
A function protecting all or part of the current paragraph. If "line" is |
| 293 |
given, it should be a number _n_, and line _n_ in the current paragraph will |
| 294 |
be protected; without "line", the entire paragraph is protected. Protecting |
| 295 |
means that the patterns not yet searched for will be ignored. For instance, |
| 296 |
if you want material to be read verbatim when surrounded with "<code>" and |
| 297 |
"</code>", you can declare a pattern as follows: |
| 298 |
|
| 299 |
local function verbatim (buffer) |
| 300 |
buffer[1] = "\\verbatim" |
| 301 |
buffer[#buffer - 1] = "\\endverbatim" |
| 302 |
intepreter.protect() |
| 303 |
end |
| 304 |
interpreter.add_pattern{ |
| 305 |
pattern = "^%s*<code>%*s$", |
| 306 |
call = verbatim, |
| 307 |
class = 1 |
| 308 |
} |
| 309 |
|
| 310 |
This code is extremely simplified : it assumes that "<code>" and "</code>" |
| 311 |
starts and ends the paragraph and that "</code>" isn't the last line of the |
| 312 |
file (otherwise it'd also be the last line in the paragraph, whereas here |
| 313 |
the last one is the paragraph boundary). An important point is that the |
| 314 |
pattern belongs to the first class, so it is called before all other |
| 315 |
patterns (provided there is no shorter pattern in class~1) and prevents them |
| 316 |
from doing anything, since the entire paragraph is protected. (Typesetting |
| 317 |
the material as verbatim material obviously depends on the "\verbatim" |
| 318 |
macro, not on Interpreter.) |
| 319 |
|
| 320 |
> interpreter.escape |
| 321 |
A character which prevents patterns from being replaced if immediately |
| 322 |
preceded by it. As an example, if "interpreter.escape = '_'", and "*text*" |
| 323 |
denotes italic, then "*text*" will produce _text_ while "_*text*" will |
| 324 |
produce *text*. Once a paragraph has been processed, Interpreter removes all |
| 325 |
escape characters. Only one character can be an escape character. |
| 326 |
|
| 327 |
> interpreter.protector(left[, right]) ["right" defaults to "left"] |
| 328 |
Defines two characters to protect what they surround. In other words, |
| 329 |
Interpreter replaces patterns only if the match isn't found between "left" |
| 330 |
and "right". Unlike the escape character, you can define as many protectors |
| 331 |
as you wish; and unlike the escape character again, Interpreter _doesn't_ |
| 332 |
remove them once the paragraph has been processed, so you must take care of |
| 333 |
them. For instance: |
| 334 |
|
| 335 |
intepreter.protector('"') |
| 336 |
interpreter.add_pattern{ |
| 337 |
pattern = '"(.-)"', |
| 338 |
replace = '\\verb`%1`', |
| 339 |
class = 0 |
| 340 |
} |
| 341 |
|
| 342 |
Anything between double quotes will be left untouched; then, when the |
| 343 |
paragraph has been processed for all other classes, a pattern in class~0 |
| 344 |
calls the "\verb" command to take care of the argument. Note that the |
| 345 |
protectors should enclose what they protect without coinciding with it; this |
| 346 |
is not the case here, which is why the pattern is applied. |
| 347 |
|
| 348 |
> interpreter.direct [Default: two percent signs then "I" and at least one space] |
| 349 |
A string, actually a pattern, signalling that the line which it begins |
| 350 |
should be processed as Lua code. The default is "%%%%I%s+", i.e. "%%I" |
| 351 |
followed by at least one space. The pattern shouldn't declare itself as |
| 352 |
attached to the beginning of the line (as in "^%%%%I%s+") because they will |
| 353 |
be matched at the beginning of the line only anyway. The line is processed |
| 354 |
with the "loadstring" function, and then turned into an empty line. For |
| 355 |
instance: |
| 356 |
|
| 357 |
%%I interpreter.active = false |
| 358 |
This won't be interpreted... |
| 359 |
%%I interpreter.active = true |
| 360 |
|
| 361 |
As this example shows, lines flagged with "interpreter.direct" don't obey |
| 362 |
"interpreter.active" and are always processed as described above. |
| 363 |
|
| 364 |
|
| 365 |
|
| 366 |
================================== technical_tag |
| 367 |
=== Technical stuff ============== |
| 368 |
================================== |
| 369 |
|
| 370 |
You don't have to bother with this section if you don't mind how Interpreter |
| 371 |
does its job; actually you won't learn much anyway. |
| 372 |
|
| 373 |
> interpreter.reset() |
| 374 |
A function which resets everything to default and deletes classes. It is used |
| 375 |
when calling "\interpretefile" so that new interpretetions start from zero. |
| 376 |
|
| 377 |
> interpreter.register(function) |
| 378 |
A function called to put Interpreter's main function into the |
| 379 |
"post_linebreak_filter" callback; you can redefine it at will. If it is |
| 380 |
undefined, "callback.register()" is used, unless "luatexbase.add_to_callback()" |
| 381 |
is detected. (The detection takes place at the first call to |
| 382 |
"\interpretfile", so there is no need to load Interpreter after |
| 383 |
"luatexbase".) |
| 384 |
|
| 385 |
> interpreter.unregister(function) |
| 386 |
A function called to remove Interpreter's main function from the |
| 387 |
"post_linebreak_filter" callback. It works similarly to the previous one. |
| 388 |
|
| 389 |
|
| 390 |
|
| 391 |
================================== example_tag |
| 392 |
=== An example: i-doc.lua ======== |
| 393 |
================================== |
| 394 |
|
| 395 |
\interpretfile{doc}{i-doc.lua} |
| 396 |
|
| 397 |
|
| 398 |
|
| 399 |
================================== gates_tag |
| 400 |
=== The Gates in Interpreter ===== |
| 401 |
================================== |
| 402 |
|
| 403 |
Interpreter is written with the Gates package (only the Lua side, |
| 404 |
actually). It means that it can be hacked down to the core. Here I'll |
| 405 |
simply list the gates involved; you should read the Gates |
| 406 |
documentation to learn how to use them. |
| 407 |
|
| 408 |
There are three gates families: "interpreter", associated with the main |
| 409 |
"interpreter" table, contains the user interface; "interpreter_tools", |
| 410 |
associated with "interpreter.core.tools" table, contains internal |
| 411 |
functions; finally "interpreter_reader", associated with the |
| 412 |
"interpreter.core.reader" table, contains the main functions used to |
| 413 |
read the file. |
| 414 |
|
| 415 |
Whenever I mention a conditional or a loop, I mean the local conditionals |
| 416 |
and loops, relative to the l-gate where the gate appears. Also, the |
| 417 |
syntax indicates the arguments a gate uses, not all the arguments that |
| 418 |
are passed to it (which are simply what the previous gate has returned). |
| 419 |
|
| 420 |
As an example of customizing Interpreter with Gates, you could very |
| 421 |
well add a bit of code which does something to all lines. Inserting a |
| 422 |
small gate, say "everyline", after "check_direct" in "aggregate_lines" |
| 423 |
below would do the trick, e.g.: |
| 424 |
|
| 425 |
|
| 426 |
function interpreter.core.reader.everyline (file, line) |
| 427 |
line = dosomething(line) |
| 428 |
return file, line |
| 429 |
end |
| 430 |
interpreter.core.reader.add( |
| 431 |
"everyline", "aggregate_lines", "after check_direct") |
| 432 |
interpreter.core.reader.conditional( |
| 433 |
"everyline", "aggregate_lines", function (f, l) return l end) |
| 434 |
|
| 435 |
(Note that it is important to check that the line really exist, |
| 436 |
because one might have hit the end of the file; hence the conditional, |
| 437 |
as with others gates in "aggregate_lines"). |
| 438 |
|
| 439 |
|
| 440 |
|
| 441 |
|
| 442 |
================================== interpreter_tag |
| 443 |
=== The "interpreter" table ======== |
| 444 |
================================== |
| 445 |
|
| 446 |
All the user functions in "interpreter" are simple m-gates, so they can |
| 447 |
be treated as ordinary functions, except "interpreter.add_pattern", |
| 448 |
which is an l-gate containing, built as: |
| 449 |
|
| 450 |
add_pattern |
| 451 |
. ensure_class |
| 452 |
. apply_nomagic |
| 453 |
. insert-pattern |
| 454 |
. . do_insert |
| 455 |
. . sort_class |
| 456 |
|
| 457 |
> ensure_class (<pattern>) [m-gate] |
| 458 |
Creates the class of <pattern> if necessary, and set it as the metatable |
| 459 |
for <pattern>. Classes themselves are kept in the "interpreter.core.classes" |
| 460 |
table. The gate return <pattern> and the class number. |
| 461 |
|
| 462 |
> apply_nomagic (<pattern>, <class>) [m-gate] |
| 463 |
Transforms the "pattern" entry in <pattern> with "intepreter.nomagic"; |
| 464 |
tied to a conditional that returns <pattern>'s "nomagic" entry (so |
| 465 |
the gate is executed only if "nomagic" is true); autoreturns both |
| 466 |
arguments. |
| 467 |
|
| 468 |
> insert_pattern (<pattern>, <class>) [l-gate] |
| 469 |
An autoreturning l-gate containing the following two gates. |
| 470 |
|
| 471 |
> do_insert (<pattern>, <class>) [m-gate] |
| 472 |
Adds <pattern> to <class> (i.e. "interpreter.core.classes[<class>]"). |
| 473 |
|
| 474 |
> sort_class (<pattern>, <class>) [m-gate] |
| 475 |
Sorts <class> with function "interpreter.core.tools.sort". This gate |
| 476 |
can be skipped to apply the patterns in the order in which they were |
| 477 |
declared. |
| 478 |
|
| 479 |
|
| 480 |
|
| 481 |
============================================= interpreter_tools_tag |
| 482 |
=== The interpreter.core.tools table ======== |
| 483 |
============================================= |
| 484 |
|
| 485 |
All the functions in the "interpreter.core.tools" all are simple m-gates. |
| 486 |
|
| 487 |
> sort (<patt1>, <patt2>) [m-gate] |
| 488 |
Returns true if the pattern in "patt1" is longer than the one in |
| 489 |
"patt2", or if they have the same length and the first ranks before |
| 490 |
the second with respect to alphabetical order. The gate is used in |
| 491 |
the "interpreter.sort_class" m-gate. |
| 492 |
|
| 493 |
> xsub (<string>, <index>, <pattern>, <replacement>) [m-gate] |
| 494 |
Returns <string> with <pattern> replaced with <replacement>, but only |
| 495 |
once, and only after <index>. |
| 496 |
|
| 497 |
> protector (<string>, <index>) [m-gate] |
| 498 |
Checks whether <index> in <string> isn't between characters declared |
| 499 |
with "interpreter.protector". If that is the case, the function returns |
| 500 |
"nil" and the index of the second protector. Otherwise, it returns |
| 501 |
<index>. |
| 502 |
|
| 503 |
> get_index (<string>, <pattern>, <index>) [m-gate] |
| 504 |
Checks whether <pattern> occurs in <string>, starting at <index>. If |
| 505 |
it does, but if <index>-1 is "interpreter.escape", calls itself with |
| 506 |
<index>+1. Otherwise, calls "interpreter.core.tools.protector" to |
| 507 |
check whether <index> is in a protected part of the string. If so, |
| 508 |
calls itself with <right>+1 instead of <index>, where <right> is the |
| 509 |
second return value of of "interpreter.core.tools.protector", i.e. |
| 510 |
it searches again after the right protector. If <index> is found, |
| 511 |
end of story, returns <index>, otherwise returns nothing. |
| 512 |
|
| 513 |
The "interpreter.core.tools" table also contains |
| 514 |
"magic_characters", a table with an entry for each magic character in Lua except |
| 515 |
`"."' and `"%"'; the values to those entries are the same characters |
| 516 |
prefixed with `"%"'. The table is used by "interpreter.nomagic" to spot |
| 517 |
and replace magic characters, with the dot and the percent sign dealt |
| 518 |
with independantly. |
| 519 |
|
| 520 |
|
| 521 |
|
| 522 |
============================================= interpreter_reader_tag |
| 523 |
=== The interpreter.core.reader table ======= |
| 524 |
============================================= |
| 525 |
|
| 526 |
Interpreter works by hooking in the "open_read_file" callback; the |
| 527 |
function registered there is the "interpreter.core.reader.input" |
| 528 |
l-gate, built as follows: |
| 529 |
|
| 530 |
input |
| 531 |
. unregister |
| 532 |
. . set_unregister |
| 533 |
. . use_unregister |
| 534 |
. open_file |
| 535 |
. set_reader |
| 536 |
|
| 537 |
> unregister (<filename>) [l-gate] |
| 538 |
Contains the following two m-gates; <filename> is received from |
| 539 |
"input", which itself receives it from the callback, i.e. that's the |
| 540 |
file that's being input (the second argument to "\interpretfile"). |
| 541 |
It is also automatically returned. |
| 542 |
|
| 543 |
> set_unregister () [m-gate] |
| 544 |
Sets the function to remove "input" from the callback, namely |
| 545 |
"interpreter.unregister"; the gate is called only if gate |
| 546 |
"interpreter.unregister" doesn't already exits. If "luatexbase" is |
| 547 |
detected, the functions there are used; otherwise, "callback.register" |
| 548 |
is used with "nil" as the second argument. |
| 549 |
|
| 550 |
> use_unregister () [m-gate] |
| 551 |
Calls "interpreter.unregister()". (You don't want the next input file |
| 552 |
to be processed with Interpreter by default, that's why you remove |
| 553 |
the callback function; not that the current one is nonetheless |
| 554 |
processed with the current file, of course.) |
| 555 |
|
| 556 |
> open_file (<filename>) [m-gate] |
| 557 |
Returns "io.open(<filename>)". |
| 558 |
|
| 559 |
> set_reader (<file>) [m-gate] |
| 560 |
Returns a table with a "reader" entry containing a function whose |
| 561 |
definition is |
| 562 |
|
| 563 |
function () |
| 564 |
return interpreter.core.reader.read_file(f) |
| 565 |
end |
| 566 |
|
| 567 |
That's the convention for the "open_read_file" callback: it should |
| 568 |
return such a table, and the function will be called each time a |
| 569 |
line is required from the input file. |
| 570 |
|
| 571 |
So most of the work is done by "interpreter.core.reader.read_file", |
| 572 |
which is why it is so heavy; it receives a file handle: |
| 573 |
|
| 574 |
read_file |
| 575 |
. make_paragraph |
| 576 |
. . aggregate_lines |
| 577 |
. . . read_line |
| 578 |
. . . check_direct |
| 579 |
. . . insert_line |
| 580 |
. . apply_classes |
| 581 |
. . . pass_class |
| 582 |
. . . . pass_pattern |
| 583 |
. . . . . process_lines |
| 584 |
. . . . . . switch |
| 585 |
. . . . . . call |
| 586 |
. . . . . . replace |
| 587 |
. . . . . . protect |
| 588 |
. . . unprotect |
| 589 |
. . . . undo_protected |
| 590 |
. . . . unprotect_lines |
| 591 |
. . . remove_escape |
| 592 |
. return_line |
| 593 |
|
| 594 |
> make_paragraph (<file>) [l-gate] |
| 595 |
The big l-gate that contains everything that follows, barring |
| 596 |
"return_line". It is called if and only if the "interpreter.core.lines" |
| 597 |
table is empty; that table is where lines of a paragraph are stored, |
| 598 |
and it is emptied by "return_line". |
| 599 |
|
| 600 |
> aggregate_lines (<file>, <line>) [l-gate] |
| 601 |
The main l-gate that reads line from <file> and stores them in |
| 602 |
"interpreter.core.lines". It loops until <line> is "nil" or equivalent |
| 603 |
to "interpreter.paragraph". (Of course, <line> is "nil" on the first |
| 604 |
iteration, but the "loopuntil" conditional is evaluated after that |
| 605 |
first iteration, during which the last subgate "insert_line" will |
| 606 |
probably returns a line.) |
| 607 |
|
| 608 |
> read_line (<file>) [m-gate] |
| 609 |
Reads the next line from <file>, and returns <file> and that line. |
| 610 |
|
| 611 |
> check_direct (<file>, <line>) [m-gate] |
| 612 |
If <line> begins with "interpreter.direct", removes it and use |
| 613 |
"loadstring" on the resulting string. Returns <file> and <line>, the |
| 614 |
latter set to an empty string is the previous operation applied. The |
| 615 |
gate depends on a conditional: <line> should be non-"nil" (of |
| 616 |
course), and "interpreter.direct" should be defined. |
| 617 |
|
| 618 |
> insert_line (<file>, <line>) [m-gate] |
| 619 |
Adds <line> to "interpreter.core.lines". Automatically returns the |
| 620 |
two arguments (and if <line> isn't "nil" or equivalent to |
| 621 |
"interpreter.paragraph", it will be executed again). |
| 622 |
|
| 623 |
> apply_classes () [l-gate] |
| 624 |
The l-gate that applies transformations to the lines, once the paragraph |
| 625 |
has been gathered, with the gates that follow. For each class, it |
| 626 |
will apply each pattern on each line. It depends on a conditional: |
| 627 |
"interpreter.core.lines" shouldn't be empty, and "interpreter.active" |
| 628 |
should be true. |
| 629 |
|
| 630 |
> pass_class () [l-gate] |
| 631 |
This gate iterates on all classes in "interpreter.core.classes" and |
| 632 |
then on class 0. On each iteration it checks beforehand whether the |
| 633 |
paragraph is protected, i.e. "interpreter.core.reader.protected" isn't |
| 634 |
a boolean (see "unprotecte" below). On each iteration, the class |
| 635 |
number and the class itself are returned. (This behavior is |
| 636 |
implemented with a Gates iterator.) |
| 637 |
|
| 638 |
> pass_pattern (<ignored>, <class>) [l-gate] |
| 639 |
Same as "pass_class", except it iterates on the patterns in "class": |
| 640 |
it is executed as long as "interpreter.core.reader.protected" and |
| 641 |
returns the patter number and the pattern itself. (The <ignored> |
| 642 |
argument isn't used; it is for the "pass_class" iterator; the same |
| 643 |
holds for the following gates.) |
| 644 |
|
| 645 |
> process_lines (<ignored>, <pattern>) [l-gate] |
| 646 |
Again, this calls an iterator. It browses each line in "interpreter.core.lines" |
| 647 |
and returns the line's number (provided it is valid, i.e. not a table, |
| 648 |
see "protect" below), <pattern> and the current index in that line. |
| 649 |
To keep track of the current line and index, two internal numbers are |
| 650 |
used: "interpreter.core.reader.current_line" and |
| 651 |
"interpreter.core.reader.current_index". |
| 652 |
|
| 653 |
> switch (<line>, <pattern>, <index>) [m-gate] |
| 654 |
If <pattern> has a "call" entry, it sets the "call" gate below to |
| 655 |
"ajar"; otherwise, if <pattern> has a "replace" entry, it sets the |
| 656 |
"replace" gate to "ajar". |
| 657 |
|
| 658 |
> call (<line>, <pattern>, <index>) [m-gate] |
| 659 |
This gate is closed by default and set to "ajar" by "switch" above. |
| 660 |
If, starting at <index>, the <pattern>'s "pattern" entry can be |
| 661 |
found in <line> with "interpreter.core.tools.get_index" (which makes |
| 662 |
sure that protectors are obeyed and returns <newindex>, where the |
| 663 |
pattern is found if it occurs), the <pattern>'s "call" entry is |
| 664 |
applied as |
| 665 |
|
| 666 |
<pattern>.call(interpreter.core.lines, |
| 667 |
<line>, <newindex>, <pattern>) |
| 668 |
|
| 669 |
This may returned zero, one or two values. If nothing is returned, |
| 670 |
"interpreter.core.reader.current_index" is set to 0, which makes |
| 671 |
the "process_lines" iterator consider the next line. If one value is |
| 672 |
returned, it is the new current index and "process_lines" will not |
| 673 |
update the line number. If two values are returned, the first is the |
| 674 |
new current line number and the second the new current index. |
| 675 |
|
| 676 |
> replace (<line>, <pattern>, <index>) [m-gate] |
| 677 |
This gate is closed by default and set to "ajar" by "switch" above. |
| 678 |
This tries to find the <pattern>'s "pattern" like "call" above, and if |
| 679 |
it is found, it applies "interpreter.core.tools.xsub" as: |
| 680 |
|
| 681 |
xsub (interpreter.core.lines[<line>], |
| 682 |
<newindex>, <pattern>.pattern, <pattern>.replace) |
| 683 |
|
| 684 |
where <newindex> is defined as in "call". The return value of "xsub" |
| 685 |
is assigned to "interpreter.core.lines[<line>]", and the current |
| 686 |
index is set to <index> plus the <pattern>'s "offset" if any. If the |
| 687 |
pattern wasn't found, the current index is set to 0, which makes |
| 688 |
"process_lines" turn to the next line as explained in "call". |
| 689 |
|
| 690 |
> protect () [m-gate] |
| 691 |
The "interpreter.protect()" function can either protect the whole |
| 692 |
paragraph (when no argument is passed) or a single line (when a number |
| 693 |
is passed). In the first case, "interpreter.core.tools.protected" |
| 694 |
takes the value "true", which is checked in various gates above. In |
| 695 |
the second case, "interpreter.core.tools.protected" is a table with |
| 696 |
each index indicating a line to be protected. This gate implements |
| 697 |
the protection in that case: it iterates on all entries in the table |
| 698 |
with "pairs" and protects the line with the same index in |
| 699 |
"interpreter.core.lines" by transforming it into a table (with a |
| 700 |
single entry, the string representing the original line); the type |
| 701 |
of the line is checked in the "process_lines" iterator above. The |
| 702 |
gate's iterator doesn't take arguments, but the function itself is |
| 703 |
defined as taking a number (the line). |
| 704 |
|
| 705 |
> unprotect () [l-gate] |
| 706 |
Now all the patterns in all the classes have been applied to the |
| 707 |
entire paragraph, and protection must be removed. This l-gate |
| 708 |
contains the following two gates. |
| 709 |
|
| 710 |
> undo_protected () [m-gate] |
| 711 |
Simply sets "interpreter.core.tools.protected" to "nil" so it is |
| 712 |
ready for the next paragraph. |
| 713 |
|
| 714 |
> unprotect_lines () [m-gate] |
| 715 |
Restores all the lines in "interpreter.core.lines" as simple strings. |
| 716 |
The gate uses an iterator that simply runs "ipairs" on "interpreter.core.lines", |
| 717 |
so the function's definition actually takes the line's number and |
| 718 |
the line itself as arguments. |
| 719 |
|
| 720 |
> remove_escape () [m-gate] |
| 721 |
If "interpreter.escape" is defined, removes all its occurrences in |
| 722 |
each line of the paragraph. |
| 723 |
|
| 724 |
This is the end of the big "make_paragraph" l-gate. It won't be called |
| 725 |
again until the paragraph has been fully passed to TeX, i.e. when |
| 726 |
"interpreter.core.lines" is empty. |
| 727 |
|
| 728 |
> return_line () [m-gate] |
| 729 |
Pops the first line from "interpreter.core.lines" and returns it. |
| 730 |
Since this is the very last subgate of "read_file", the line is |
| 731 |
passed to TeX. |
| 732 |
|