Debian Codegraph @

texmf-dist/doc/generic/tokstools/tokstools-en.tex

raw
1 % !TeX TS-program = lualatex
2 % !TeX encoding = UTF-8
3 % This is an AI-generated translation into English of the “toktools” package manual
4 \documentclass[english,a4paper,10pt]{article}
5 \usepackage[margin=2cm]{geometry}
6 \usepackage[bottom]{footmisc}
7 \usepackage[libertine]{newtxmath}
8 \usepackage{libertineRoman,enumitem,babel,xspace,tokstools,hyperref,array,footnotehyper,multicol}
9 \makesavenoteenv{tabular}
10 \RequirePackage[most]{tcolorbox}
11 \tcbuselibrary{listings}
12 \usepackage[scaled=0.8]{GoMono}
13 \def\tktl{\texttt\tktlname\xspace}
14
15 \catcode`\_11
16 \long\def\def_active#1#2{%
17 \catcode`#1=13
18 \begingroup
19 \lccode`\~`#1
20 \lowercase{\endgroup\long\def~{#2}}%
21 }
22
23 %%%%%%%%%%%%% la macro \tktlcode %%%%%%%%%%%%%
24 \def\nolig_list{\do\`\do\<\do\>\do\,\do\'\do\-}
25 \def\do_noligs#1{%
26 \catcode`#1\active
27 \begingroup
28 \lccode`\~`#1\relax
29 \lowercase{\endgroup\def~{\leavevmode\kern0pt \char`#1\relax}}%
30 }
31 \def\no_ligs{\let\do\do_noligs \nolig_list}
32 \newif\ifspecialcode
33 \specialcodefalse
34 \def\specialcode{\specialcodetrue}
35 \def\tktlcode{\expandafter\tktlcode_i\string}
36 \def\tktlcode_i#1{%
37 \begingroup
38 \nobreak\smallbreak
39 \parindent0pt
40 \parskip=0pt
41 \leftskip2.5em
42 \rightskip0pt
43 \nonfrenchspacing
44 \no_ligs
45 \def\do##1{\catcode`##1=12\relax}\dospecials
46 \ifspecialcode\else
47 \def_active\<{\begingroup$\langle$\ttfamily\itshape}%
48 \def_active\>{$\rangle$\endgroup}%
49 \def_active\&{\string}%
50 \def_active\_{\tktlsubscript}%
51 \fi
52 \ttfamily
53 \def_active\^^I{\space\space\space\space}%
54 \def_active\^^M{\par\noindent}%
55 \def_active\ {\space}%
56 \catcode`#1=12
57 \def\tktlcode_ii##1#1{##1\par\endgroup\specialcodefalse}%
58 \tktlcode_ii
59 }
60 \catcode`\_8
61 \def\tktlsubscript#1{$_{#1}$}
62
63 \makeatletter
64 \let\Verb\verb
65 \begingroup
66 \catcode`\<13 \catcode`\>13
67 \gdef\verb{\relax\ifmmode\hbox\else\leavevmode\null\fi
68 \bgroup
69 \verb@eol@error \let\do\@makeother \dospecials
70 \verbatim@font\@noligs
71 \catcode`\<13 \catcode`\>13 \def<{\begingroup$\langle$\itshape}\def>{\/$\rangle$\endgroup}%
72 \@ifstar\@sverb\@verb}
73 \endgroup
74 \def\longfrdate@i#1/#2/#3\@nil{\number#3\relax\space \ifcase#2 \or janvier\or février\or mars\or avril\or mai\or juin\or juillet\or aout\or septembre\or octobre\or novembre\or décembre\fi\space#1}
75 \edef\longfrdate{\expandafter\longfrdate@i\tktldate\@nil}
76 \def\<#1>{$\langle${\ttfamily\itshape#1\/}$\rangle$}
77 \makeatother
78 \newtcblisting{doclatex}{
79 boxrule=.5pt,
80 boxsep=0pt,
81 top=5pt,
82 left=5pt,
83 right=5pt,
84 bottom=5pt,
85 skin=bicolor,
86 colbacklower = white,
87 colback = black!15,
88 listing options={%
89 style=tcblatex,
90 language={TeX},
91 showspaces=false,
92 showstringspaces=false,
93 commentstyle=\itshape\color{gray!80!white},
94 basicstyle=\ttfamily\footnotesize
95 }
96 }
97
98 \begin{document}
99 \parindent=0pt
100 \thispagestyle{empty}
101 \begin{titlepage}
102 \begingroup
103 \centering
104 \null\vskip.25\vsize
105 {\large\bfseries The package for \TeX{} and \LaTeX\par \Huge \tktlname\par}
106 \bigbreak
107 v \tktlver
108 \smallbreak
109 \tktldate
110 \vskip1.5cm
111 Christian {\scshape Tellechea}\par
112 \texttt{[email protected]}\par
113 \endgroup
114 \vskip2cm
115 \leftskip=.2\linewidth \rightskip=.2\linewidth \small
116 This extension for \TeX{} or \LaTeX{} provides tools based on Parsing Expression Grammars (PEGs) for manipulating tokens. Given an arbitrary set of tokens with balanced braces, it is possible to define matching rules to test for matches, count them, perform replacements, capture tokens, and more.
117 \end{titlepage}
118 \parindent0pt \pagestyle{plain}
119 \tableofcontents
120 \parskip\medskipamount
121 \newpage
122
123 \section{Presentation}
124 The \tktl extension works with any engine. It is not limited to \LaTeX{}, as the \texttt{pegmatch} package unfortunately is, and to use it, you need to write
125 \begin{itemize}
126 \item \verb|\input tokstools.tex| with (pdf)(Xe)(lua)(op)\TeX\ (the syntax \verb|\input{tokstoosl}| is also accepted);
127 \item \verb|\usepackage{tokstools}| with (pdf)(Xe)(lua)\LaTeX.
128 \end{itemize}
129
130 This package only requires \href{https://ctan.org/pkg/simplekv}{simplekv}, which will be loaded if it hasn't been.
131
132 \textit{Important notice}: this English manual is provided so that as many people as possible can access the necessary instructions for using the \tktl package. It is very important for readers of this manual to understand that it is merely a translation generated by various AI-based tools available online. The quality and accuracy of the translation are therefore not guaranteed, especially since the resulting translation has not been reviewed by a human. Consequently, \textbf{for reliable information, please refer to the French manual}.
133 %
134 %
135 %
136 %
137 %
138 %
139 %
140 %
141 %
142 %
143 %
144 %
145 %
146
147 \subsection{What’s new in version 0.2}
148 \paragraph{Macro \texttt{\textbackslash pegreplace}}
149 The \verb|\pegreplace| macro has a new syntax while offering additional features; see page~\pageref{pegreplace}. Unfortunately, this change breaks the syntax from version 0.1. I sincerely apologize to \tktl users for this inconvenience.
150
151 \paragraph{Macro \texttt{\textbackslash printtoks}}
152 The \verb|\printtoks| macro can now, depending on the value of the new \verb|mode| key, redirect token information to the log file; see page~\pageref{printtoks}.
153
154 \paragraph{Validity of assignment directives}
155 From now on, no validity checks are performed on the “assignment directives” passed via the \verb|assign| key and similar mechanisms. The user must therefore make an informed choice between:
156 \begin{itemize}
157 \item \verb|assign={}|, which is an empty directive to send the result to the \TeX{} input stream for typesetting;
158 \item \verb|assign=\def<macro>| so that the result is assigned to the \<macro>;
159 \item \verb|assign=<token register>| to assign the result to a token register, which is necessary when it contains tokens of catcode 6 (\verb|#|) to denote arguments.
160 \end{itemize}
161
162 \subsection{Tokens, charcode, catcode, engines}
163 The \tktl package operates on sets of tokens (or token strings). Tokens are the smallest unit of source code manipulated by \TeX.
164
165 Users should be aware that the notion of tokens depends on the type of engine used. It is only for 8~bit engines, such as pdf\LaTeX{} for example, that \hbox{1 token${}={}$1 byte}. As an example, the character “\verb|€|” is seen
166 \begin{itemize}
167 \item by pdf\LaTeX{} as 3 tokens whose charcodes are 226; 130 and 172 and catcodes are all equal to 13;
168 \item by lua\LaTeX{} as 1 single token with charcode 8364 and catcode 12.
169 \end{itemize}
170
171 This difference must be taken into account, and can be the source of more or less understandable errors when manipulating UTF8 chars on an 8~bit engine.
172
173 This manual was compiled with lua\LaTeX, so tokens with charcodes greater than 255 may exist.
174
175 Whatever macros are provided by \tktl, they always decompose their token argument, often called \<tokens>, into a list of tokens, each with a charcode and a catcode. The catcodes visible to \tktl are listed in the table below, along with the most common tokens with charcodes less than 255 that have these catcodes by default.
176 \begin{center}
177 \renewcommand\arraystretch{1.15}
178 \begin{tabular}{>{\leftskip0ptplus1fill \rightskip\leftskip}p{1.25cm}p{5.5cm}p{6.5cm}}\hline
179 Catcode & Description & Tokens\\\hline
180 1 & beginning of a group & \verb|{|\\
181 2 & end of a group & \verb|}|\\
182 3 & math shift & \verb|$|\\
183 4 & alignment tab & \verb|&|\\
184 6 & parameter character & \verb|#|\\
185 7 & superscript & \verb|^|\\
186 8 & subscript & \verb|_|\\
187 10 & space & \Verb*| | and \itshape HT\\
188 11 & letters & \verb|a-z| and \verb|A-Z|\\
189 12 & others & digits (\verb|0-9|)\par
190 punctuation signs (\verb|: . ; , ? ! ' "|)\par
191 math signs (\Verb|+ - * / = < >|)\par
192 other signs (\verb/[ ] ( ) | @ `/)\\
193 13 & active character & \verb|~|\\
194 16 & control sequence & \verb|\<letters>| ou \verb|\<char>| \\
195 \hline
196 \end{tabular}
197 \end{center}
198 None of the macros provided by \tktl are expandable. If you wish the results of these macros, or even captures, to be contained in macros that are purely expandable, you must use the assignment instructions defined in this manual.
199
200 \subsection{Notations}
201 What is referred to as a “marker” in this manual is a control sequence (or character) that acts as a reference point to be recognized within an argument, but which is not modified by \tktl and whose meaning is irrelevant.
202
203 Throughout this manual:
204 \begin{itemize}
205 \item \<tokens> denotes a set of tokens\footnote{The primitive \texttt{\textbackslash par} may be found in \<tokens>.} balanced in tokens of catcode 1 and 2 (opening and closing braces) that serves as an argument to macros or markers in charge of processing these tokens;
206 \item \<1-pattern> represents a single pattern, i.e., consisting of a marker with its possible argument, preceded by an optional predicate and followed by an optional repetition specification;
207 \item \<patterns> represents a single \<1-pattern> or a combination of several \<1-pattern> obtained using the concatenation operator «\verb|:|» and the ordered choice operator «\verb/|/» (only the “\verb/|/” operator for \verb|\toksdo| and \verb|\tokscount|);
208 \item \verb|{<patterns>}| is a group that accepts an optional predicate and repetition directive and behaves like a \<1-pattern>.
209 \end{itemize}
210
211 \subsection{Information about tokens using {\ttfamily\textbackslash printtoks}}\label{printtoks}
212 The macro
213 \tktlcode|\printtoks[<keys>=<values>]{<tokens>}|
214 displays tokens with their charcode and catcode below. If a token is a control sequence or a control character, no charcode is displayed.
215
216 \begin{doclatex}
217 \catcode`\!=6 % “!” is similar to “#”
218 \printtoks[code=\ttfamily]{\def\foo#1!2#3{\hbox to 1cm{\hss$!1^#2_!3$\hss}}}
219 \medbreak
220 \printtoks[code=\ttfamily]{eéė€}
221 \end{doclatex}
222
223 The \<keys> available are :
224 \begin{center}
225 \renewcommand\arraystretch{1.25}
226 \begin{tabular}{>\ttfamily p{3.5cm}>\ttfamily p{2.5cm}p{8cm}}\hline
227 \rmfamily Key & \rmfamily Default value & Description\\\hline
228 expand arg & 0 & number of expansions for the argument containing the \<tokens> before being taken into account\\
229 code & \<empty> & code executed before displaying a token\\
230 intertoks & 0.33em & horizontal space between each token, inserted by the primitive \verb|\hskip|\\
231 printcharcode & true & display or not the charcode\\
232 printcatcode & true & display or not the catcode\\
233 hexcharcode & false & display the charcode in base 16 if \verb|true| and in base 10 otherwise\\
234 baselinecoeff & 0.8 & vertical spacing coefficient\\
235 vlines & true & display or not the vertical lines between each token\\
236 boxed & true & put everything in a \verb|\hbox| if \verb|true|, which makes everything unbreakable\\
237 mode & 0 & If 1 or 2, redirect token information to the log file\\\hline
238 \end{tabular}
239 \end{center}
240
241 When the \verb|mode| key is set to 1, information about each token is also written to the log file. When this option is set to 2, the information is written only to the log file, and nothing is displayed in the PDF.
242
243 For example, the following code
244 \specialcode\tktlcode|\def\foo#1{$#1$}
245 \catcode`\!=6 % “!” is similar to “#”
246 \printtoks[mode=2]{\def\foo#1!2#3{\hbox to 1cm{\hss$!1^#2_!3$\hss}}}|
247 generates the following lines in the log file:
248
249 \columnseprule0.4pt
250 \begin{multicols}2
251 \specialcode\tktlcode|------ Begin token list ------
252 \def catcode 16 (primitive)
253 \foo catcode 16 (macro:#1->$#1$)
254 # catcode 6 (macro parameter charcater)
255 1 catcode 12 (character)
256 ! catcode 6 (macro parameter charcater)
257 2 catcode 12 (character)
258 # catcode 6 (macro parameter charcater)
259 3 catcode 12 (character)
260 { catcode 1 (begin-group character)
261 \hbox catcode 16 (primitive)
262 t catcode 11 (letter)
263 o catcode 11 (letter)
264 catcode 10 (blank space)
265 1 catcode 12 (character)
266 c catcode 11 (letter)
267 m catcode 11 (letter)
268 { catcode 1 (begin-group character)
269 \hss catcode 16 (primitive)
270 $ catcode 3 (math shift character)
271 ! catcode 6 (macro parameter charcater)
272 1 catcode 12 (character)
273 ^ catcode 7 (superscript character)
274 # catcode 6 (macro parameter charcater)
275 2 catcode 12 (character)
276 _ catcode 8 (subscript character)
277 ! catcode 6 (macro parameter charcater)
278 3 catcode 12 (character)
279 $ catcode 3 (math shift character)
280 \hss catcode 16 (primitive)
281 } catcode 2 (end-group character)
282 } catcode 2 (end-group character)
283 ------- End token list -------|
284 \end{multicols}
285
286 \section{Act on each token with \texttt{\textbackslash toksdo}}
287 This macro scans all tokens one by one, independently of each other. The user can specify criteria using \<patterns> and for each, an \<action> to be performed if the token matches \<patterns>.
288
289 \subsection{Elementary patterns}
290 3 elementary patterns likely to match a token are available, each accessible via the marker \verb|\r|, \verb|\R| and \verb|\S|.
291
292 \subsection{The pattern \texttt{\textbackslash r}}\label{motif:r}
293 The syntax of this \<1-pattern> is
294 \tktlcode/\r{<csv of token intervals>: <csv of catcode intervals>}/
295 where an interval is of the form “range extending between two tokens” of type \verb|<token1>-<token2>|, but can be reduced to a single token \verb|<token>|.
296
297 In the \<csv list of catcode intervals> can be replaced by the character “\verb|*|”, which acts as a wildcard and stands for any interval. If the \<csv of catcode intervals> is omitted, it is understood as “\verb|*|”.
298
299 Commas, hyphens and “\verb|:|” cannot be specified as tokens, as they are part of the syntax (see the \verb|R| pattern to get around this limitation).
300
301 For example:\par
302 \begin{tabular}{>\raggedleft p{7cm}p{9cm}}
303 \verb|\r{a-z:11}| & matches with a token from \verb|a| to \verb|z| with an 11 catcode.\\
304 \verb|\r{a-z}| ou \verb|\r{a-z:*}| & matches with a token from \verb|a| to \verb|z| with any catcode.\\
305 \verb|\r{a,e,i,o,u,y}| & matches with any voyel with any catcode.\\
306 \verb|\r{a-z,A-Z,0-9:11,12}| & matches any alphanumeric character (lowercase, uppercase or numeric), with either an 11 or 12 catcode.
307 \end{tabular}
308
309 It is important to note that \emph{no extra spaces} are allowed in \<csv> syntax:
310 \begin{itemize}
311 \item \Verb*|\r{ a-z }| causes a compilation error: «\texttt{! Improper alphabetic constant}»; you must write \verb|\r{a-z}|;
312 \item \Verb*|\r{1, ,2 }| is also incorrect; you must write \Verb*|\r{1, ,2}|.
313 \end{itemize}
314
315 \subsection{The pattern \texttt{\textbackslash R}}\label{motif:R}
316 The syntax of this \<1-pattern> is
317 \tktlcode/\r{<csv of charcode intervals>:<csv of catcode intervals>}/
318
319 In both csv lists, the “\verb|*|” character is a wildcard and replaces any interval. The charcodes are numbers written in digits (base 10) or in the manner of \TeX, where “\verb|`\:|”, “\verb|`\,|” and “\verb|`\-|” are the integers corresponding respectively to the charcodes of the “\verb|:|”, “\verb|,|” and “\verb|-|” tokens.
320
321 For example:\par
322 \begin{tabular}{>\raggedleft p{6cm}p{9cm}}
323 \verb|\R{*:10}| & matches any catcode token 10 (which is a space for \TeX)\\
324 \verb|\R{*:16}| & matches any catcode token 16 (which is a character or control sequence for \TeX)\\
325 \verb|\R{106-115: 11}| & matches with any token from “\verb|j|” to “\verb|s|” having a catcode of 11\\
326 \verb|\R{`\a-`\z}| & is identical to \verb|\R{97-122}| which is equivalent to \verb|\r{a-z}|\\
327 \verb|\R{`\,,`\;,`\: :12}| & is identical to \verb|\R{44,59,58:12}| and matches \verb|,| or \verb|;| or \verb|:| with catcode 12\\
328 \verb|\R{*:*}| & matches any token
329 \end{tabular}
330
331 \subsection{The pattern \texttt{\textbackslash S}}\label{motif:S}
332 The syntax of this \<1-pattern> is
333 \tktlcode/\S{<tokens>}/
334
335 A token matches if it appears in \<tokens> passed as an argument to \verb|\S|.
336
337 In the tokens “\verb|ab \foo{0 123}c d|”, those that match the pattern “\verb|\S{1{a }\foo}|” are boxed in red\label{encadrerouge}:
338 \begin{center}
339 \fboxsep=1.5pt
340 \toksdo{ \S{1{a }\foo} -> \ifnum\selfcatcode=10 % si espace
341 \addtok{{\color{red}\fbox{\strut\textvisiblespace}}}%
342 \else
343 \setcatcode{12}%
344 \addtok{{\ttfamily\color{red}\fbox{\strut\self}}}%
345 \fi,
346 \R{*:*} -> \addtok{\,{\ttfamily\self}\,}
347 }{ab \foo{0 123}c d}
348 \end{center}
349
350 The \<tokens> may contain one or more nonnested occurrences of the marker \verb|\c|\label{marqueur:c:catcode}. The syntax of this marker is
351 \tktlcode/\c{<catcode>}{<tokens>}/
352
353 In the argument of \verb|\S|, the action of \verb|\c| is to assign the \<catcode> passed in the first argument to the \<tokens>. If the \<catcode> is reduced to a single digit, the curly braces are optional. The same applies to the second argument if it contains a single token.
354
355 If a token is a \<macro>, the only catcode it can receive is 12, in which case catcode 12 tokens from \verb|\string<macro>| will be created.
356
357 The catcodes accepted in the first argument of \verb|\c| are: 1, 2, 3, 4, 6, 7, 8, 10, 11, 12 and 13. The user should be extremely careful about the consequences of changing catcodes to the sensitive categories numbered 1, 2 and 6.
358
359 If we write
360 \tktlcode/\S{12\c{12}\bar3abc\c{11}{4 5.6}7\c78 9}/
361 then the \<set of tokens> contained in the \verb|\S| argument is:
362 \begin{center}
363 \catcode`\_11
364 \tktl_catcode_string{12\c{12}\bar3abc\c{11}{4 5.6}7\c78 9}
365 \expandafter\tktl_decode_enctoks\expandafter{\the\tktl_string_toks}{\printtoks[intertoks=0.75em]}
366 \end{center}
367
368 \subsection{Patterns}
369 The \<patterns> listed for \verb|\toksdo| consist of a single basic \<1-pattern> or multiple patterns separated by «\verb-|-», which means «or»:\par
370 \begin{tabular}{>\raggedleft p{7cm}p{9cm}}
371 \verb/\r{a-z:11} | \R{*:12}/ & matches any token that is a letter from \verb|a| to \verb|z| with catcode 11 or any token with catcode 12 \\
372 \verb/\R{*:16} | S{01223456789.}/ & matches any token that is a macro, a digit, or a period\\
373 \verb/\r{0-9:12} | \r{a-z,A-Z:11} | \R{*:10}/ & matches any token with the default catcode being a digit, a letter, or a space
374 \end{tabular}
375
376 \subsection{Using \texttt{\textbackslash toksdo}}\label{toksdo}
377 This macro is used as follows:
378 \tktlcode/\toksdo[<keys>=<values>]
379 {
380 <patterns_1> -&> <code_1>,
381 <patterns_2> -&> <code_2>,
382 etc.
383 <patterns_n> -&> <code_n>
384 }{<tokens>}/
385 You can specify as many \<patterns> and associated \<codes> as you wish.
386
387 Each \<code> is an arbitrary code\footnote{If this \<code> contains a comma that is not enclosed in curly braces, you must use the syntax \<patterns>\texttt{->\char`\{ }\<code>\texttt{\char`\}}.} that can modify the \<token> that matched \<patterns>. Within this \<code>, the following macros can be used:
388 \begin{itemize}
389 \item \verb|\tokslen|, which is the total number of tokens;
390 \item \verb|\selfindex|, \verb|\selfcharcode|, and \verb|\selfcatcode|, which, for the matched token, represent its index (starting at 1 and ending at \verb|\tokslen|), its charcode, and its catcode. If the token is a macro, \verb|\selfcharcode| is \emph{not} a number and is equal to the macro itself;
391 \item \verb|\addtok{<code>}| which specifies how to add the matched token to the internal collector that gathers them all for display or assignment at the end of the process;
392 \item \verb|\deltok| deletes the matched token;
393 \item \verb|\setcharcode{<arithmetic expression>}| and \verb|\setcatcode{<arithmetic expression>}| change the charcode and catcode of the matched token. It is impossible to change the catcode of a macro, and for all other tokens, the catcodes accepted by \verb|\setcatcode| are 1, 2, 3, 4, 6, 7, 8, 10, 11, 12, and 13. Special care must be taken regarding what happens if a catcode is changed to 1, 2, or 6.
394 \end{itemize}
395
396 The \<keys> accepted by the \verb|\toksdo| macro are:
397 \begin{center}
398 \renewcommand\arraystretch{1.25}
399 \begin{tabular}{>\ttfamily p{3.5cm}>\ttfamily p{2.5cm}p{8cm}}\hline
400 \rmfamily Key & \rmfamily Default value & Description\\\hline
401 expand arg & 0 & number of expansions for the argument containing the \<tokens> before being processed\\
402 collect & true & collects all selected tokens for the purpose of displaying or storing them.
403 No collection is performed if this boolean is \verb|false|\\
404 assign & \<empty> & assignment instruction : an assignment instruction is any code that may appear before the result enclosed in curly braces and the code that is executed is therefore \verb|<assignment>{<result>}|
405 If \<empty>, displays the result. If the value is a \<macro>, this \<macro> must be a token register that will receive the result.
406 The value can also be of the form \verb|\def<macro>|, where the \verb|<macro>| will receive the result\\\hline
407 \end{tabular}
408 \end{center}
409
410 Here's how to program Rot13:
411 \begin{doclatex}
412 \toksdo{
413 \r{a-m,A-M} -> \setcharcode{\selfcharcode + 13},
414 \r{n-z,N-Z} -> \setcharcode{\selfcharcode - 13}
415 }{Two plus two equals four}
416 \end{doclatex}
417
418 In this example, any digit or hyphen is replaced with “\verb|x|”, and any space is replaced with “\verb|_|” (catcode 12):
419 \begin{doclatex}
420 \toksdo{
421 \r{0-9} | \S{-} -> \setcharcode{`x},
422 \R{*:10} -> \setcharcode{`\_}\setcatcode{12}
423 }{Your password is: \textbf{758-457-384}, don't forget it!}
424 \end{doclatex}
425
426 Keep only the first half of the tokens:
427 \begin{doclatex}
428 \toksdo{\R{*:*} -> \ifnum\selfindex>\numexpr\tokslen/2\relax \deltok \fi}{123abcd689}
429 \end{doclatex}
430
431 It is sometimes necessary to assign the result to a token register in case the tokens obtained contain the catcode 6 token (\verb|#| by default):
432 \begin{doclatex}
433 \newtoks\myresult
434 \toksdo[assign=\myresult]
435 {\S{2} -> \deltok }% remove all "2"
436 {\def\foo#1{02#120}}
437 \the\myresult% execution of \def\foo#1{0#10}
438 \foo{A}, \foo{xYz}
439 \end{doclatex}
440 If we had written “\verb|assign = \def\mymacro|”, the code that would have been executed behind the scenes is
441 \tktlcode/\def\mymacro{\def\foo#1{0#10}}/
442 This code is illegal in \TeX{} (error of the type “\verb|Illegal parameter number in definition of \mymacro|”).
443
444 The \verb|collect| keyword, when set to \verb|false|, indicates that tokens should not be collected. This makes sense if the code processing the matched tokens does not modify them. Here's how to count vowels using a counter:
445 \begin{doclatex}
446 \newcount\voyel \voyel=0
447 \toksdo[collect=false]{ \S{aeiouy} -> \advance\voyel 1 }{happy texing}<\the\voyel>
448 \end{doclatex}
449
450 The argument of \verb|\setcharcode| and \verb|\setcatcode| is evaluated using the \verb|\numexpr| primitive; therefore, any arithmetic expression accepted by this primitive is valid. Here, all vowels are capitalized:
451 \begin{doclatex}
452 \toksdo{ \S{aeiouy} -> \setcharcode{\selfcharcode + `A - `a} }{Two plus two equals four}
453 \end{doclatex}
454
455 Here's how to extract the tokens located in the highest level of nested braces in two steps:
456 \begin{doclatex}
457 \newcount\nestcnt \nestcnt=0 % count level of nesting
458 \newcount\maxnestcnt \maxnestcnt=0 % is the highest level of nesting
459 \def\mycode{12{34{5{67}8}}9{{ab}c}{{d{e}f}}g}
460 \toksdo[expand arg=1,collect=false]{
461 \R{*:1} -> \advance\nestcnt1 \ifnum\nestcnt>\maxnestcnt \maxnestcnt=\nestcnt \fi,
462 \R{*:2} -> \advance\nestcnt-1
463 }{\mycode}Max nesting level = \the\maxnestcnt\par
464 Most nested tokens:
465 \toksdo[expand arg=1]{
466 \R{*:1} -> \advance\nestcnt1 \deltok,% delete "{"
467 \R{*:2} -> \advance\nestcnt-1 \deltok,% delete "}"
468 \R{*:*} -> \ifnum\nestcnt<\maxnestcnt \deltok \fi% delete everything except at the highest nesting level
469 }{\mycode}
470 \end{doclatex}
471
472 \subsection{Using \texttt{\textbackslash toksdo} and \texttt{\textbackslash addtok}}\label{addtok}
473 When the \verb|collect| key is set to \verb|true|, tokens are collected. Once they have been processed and optionally modified by \verb|\setcharcode| or \verb|\setcatcode|, each is added as-is to the internal collector, which is called upon at the end of the process to display the tokens or assign them if the \verb|assign| key specifies this.
474
475 It is possible to modify how tokens are added to the internal collector via the \verb|\addtok<code>}| macro, where \verb|<code>| is an arbitrary code in which \verb|\self|\label{marqueur:self} represents the token itself.
476
477 By default, and at the beginning of each code block following “\hbox{\verb|<patterns>|\Verb| -> |}”, the \verb|\addtok| macro is initialized to its default value:
478 \tktlcode|\addtok{\self}|
479 which means that each token must be added as-is.
480
481 If we write \verb|\addtok{\self\self}|, the matched tokens are duplicated. With \verb|\addtok{\fbox{\self}}|, they are framed using the \verb|\fbox| macro from \LaTeX.
482
483 The \verb|\deltok| macro is equivalent to \verb|\addtok{}|.
484
485 In most cases, \verb|\self| represents a single token: the one currently being processed. It is \emph{only} when a macro is detokenized (assigning 12 to its catcode) that \verb|\self| represents multiple tokens.
486
487 In this example, each macro is de-tokenized and boxed, and all spaces are replaced by “\Verb*| _ |”:
488 \begin{doclatex}
489 \fboxsep=2pt
490 \toksdo{ \R{*:16} -> \setcatcode{12}\addtok{\fbox{\self}},% detokenize and \fbox
491 \R{*:10} -> \addtok{ \string_ }
492 }{a b\foo12. 3\baz- 9}
493 \end{doclatex}
494
495 Each 1 is doubled, and each 0 is replaced with a space:
496 \begin{doclatex}
497 \toksdo{ \S{1} -> \addtok{\self\self},
498 \S{0} -> \setcatcode{10}% also possible: \addtok{ }
499 }{10110101}
500 \end{doclatex}
501
502 The macros \verb|\selfindex|, \verb|\selfcharcode|, and \verb|\selfcatcode| must \emph{not} be used within the argument of \verb|\addtok|, as they will be added as-is to the internal collector. They will only be expanded at runtime, at which point they will contain the data from the last token. If you wish to perform tests on the index, charcode, or catcode of the token to be added, you must do so outside the argument of \verb|\addtok|. Here is the source code, the result of which can be seen on page~\pageref{encadrerouge}:
503 \begin{doclatex}
504 In the tokens “\verb|ab \foo{0 123}c d|”, those that match
505 the pattern “\verb|\S{1{a }\foo}|” are boxed in red:\par
506 \hfill
507 \fboxsep=1.5pt
508 \toksdo{ \S{1{a }\foo} -> \ifnum\selfcatcode=10 % if space
509 \addtok{{\color{red}\fbox{\strut\textvisiblespace}}}%
510 \else% if not a space
511 \setcatcode{12}% detokenize
512 \addtok{{\ttfamily\color{red}\fbox{\strut\self}}}% frame red
513 \fi,
514 \R{*:*} -> \addtok{\,{\ttfamily\self}\,}% for other token add thin space
515 }{ab \foo{0 123}c d}
516 \hfill\null
517 \end{doclatex}
518
519 \section{Counting tokens with {\ttfamily\textbackslash tokscount}}\label{tokscount}
520 The macro
521 \tktlcode/\tokscount[<keys>=<values>]{<patterns>}{<tokens>}/
522 counts how many tokens match the \<patterns> in the \<tokens>. If no patterns are specified (empty argument), the macro counts all tokens.
523
524 The available \<keys> are as follows:
525 \begin{center}
526 \renewcommand\arraystretch{1.25}
527 \begin{tabular}{>\ttfamily p{3.5cm}>\ttfamily p{2.5cm}p{8cm}}\hline
528 \rmfamily Key & \rmfamily Default value & Description \\\hline
529 expand arg & 0 & number of expansions for the argument containing the \<tokens> before being taken into account\\
530 assign & \<empty> & assignment directive. If \<empty>, displays the number of tokens that matched. Otherwise, must contain an assignment statement of the type
531 \verb|\def<macro>| to assign the result to the \<macro>\\
532 assign match & \<empty> & assignment directive. If \<empty>, the matched tokens are not collected. Otherwise, must contain an assignment statement
533 \verb|\def<macro>| to assign them to a macro or \verb|<macro>| to assign them to a token register\\\hline
534 \end{tabular}
535 \end{center}
536
537 The \<patterns> accepted by \verb|\tokscount| are the same as those for \verb|\toksdo|.
538
539 \begin{doclatex}
540 1) \tokscount{ \r{a-z} | \S{10} }{ab1023truc098}\quad
541 2) \tokscount[assign=\def\cc]{ \R{*:16} }{ab\x123truc\zzz0\yy98}<\cc>\quad% counts macros
542 3) \tokscount[assign=\def\cc,assign match=\def\xx]{ \S{01} }{1{a01}1{0{b100}}1}<\cc><\xx>\quad% counts 0 and 1
543 4) \tokscount{ \R{*:12} | \S{\foo\bar\zid} }{ab\foo c12d*-ef}\quad% counts tokens "letters" or macros \foo\bar\zid
544 5) \tokscount{}{12{34}5}% counts all tokens
545 \end{doclatex}
546
547 \section{PEG Grammars}
548 In a set of tokens, some may match patterns that can be organized into a PEG (Parsing Expression Grammar). Whenever a match occurs with a pattern, the tokens that matched are consumed and are no longer available for subsequent patterns (except for predicates; see below). If there is no match, no tokens are consumed.
549
550 \subsection{The 5 Pattern Markers}\label{motifs}
551 The \tktl package provides 5 basic patterns defined by 5 markers for constructing more complex patterns:
552
553 \begin{itemize}
554 \item the pattern \verb|\r{<csv character ranges>:<csv catcode ranges>}| described on page~\pageref{motif:r};
555 \item the pattern \verb|\R{<csv charcode ranges>:<csv catcode ranges>}| described on page~\pageref{motif:R};
556 \item the pattern \verb|\S{<tokens>}| described on page~\pageref{motif:S};
557 \item the pattern \verb|\s{<tokens>}|\label{motif:s}: any set of tokens consisting exactly of the \<tokens> passed as an argument to \verb|\s| matches this pattern: this is therefore an exact match between strings of tokens.\par Just as with the \verb|\S| marker, it is possible to change the catcodes of certain tokens using the \verb|\c| marker according to the syntax \verb|\c{<catcode>}{<tokens>}|;
558 \item the pattern \verb|\.|\label{motif:dot} which matches any token and is equivalent to \verb|\R{*:*}|.
559 \end{itemize}
560
561 \subsection{Repetitions}
562 Each \<1-pattern> can be followed by a repetition marker if you wish to specify the number of times the match occurs. The repetition markers are:
563 \begin{itemize}
564 \item \verb|^<digit>| or \verb|^{<number>}|: requires that the match be repeated exactly the number of times specified as an argument to \verb|^|;
565 \item \verb|^{<min>-<max>}|: specifies that the match must occur between \verb|<min>| and \verb|<max>|. If \verb|<min>| is omitted, it is taken as 0. If \verb|<max>| is omitted, there is no upper limit on the number of repetitions. Both numbers cannot be omitted at the same time.
566 \item “\verb|+|”: 1 or more repetitions (equivalent to \verb|^{1-}|);
567 \item “\verb|*|”: 0 or more repetitions (equivalent to \verb|^{0-}|);
568 \item “\verb|?|”: 0 or 1 repetition (equivalent to \verb|^{0-1}|);
569 \end{itemize}
570
571 If no repetition marker is present after a \<1-pattern>, the directive \verb|^1| is implicitly applied.
572
573 It is important to note that \emph{in all cases}, the maximum possible number of repetitions is consumed; this behavior is always “greedy”.
574
575 \subsection{Predicates}
576 Each \<1-pattern> can be preceded by the character “\verb|!|” or “\verb|&|”, which turns it into a predicate:
577 \begin{itemize}
578 \item The predicate \verb|!<1-pattern>| matches if there is no match with the \<1-pattern>
579 \item The predicate \verb|&<1-pattern>| matches if there is a match with the \<1-pattern>.
580 \end{itemize}
581
582 The rule in PEG grammars is that \emph{a predicate never consumes tokens}: a predicate can therefore be understood as a lookahead i.e. a test on what follows without changing the position.
583
584 \subsection{Pattern concatenation}
585 The character “\verb|:|” between two \<1-pattern> indicates that the first \emph{and} the second should match. Of course, we are not limited to two patterns; there can be as many as desired.
586
587 \begin{tabular}{>\raggedleft p{6cm}p{9cm}}
588 \verb|&\r{0-9}^{3-} : \r{0-9}^2| & if there are at least 3 digits, matches the first 2\\
589 \verb|\r{a-z}+ : \R{*:10}? : \S{01}^2| & matches one or more letters followed by an optionnal space followed by 2 binary digits\\
590 \verb|\r{1-9}+ : \s{00} : !\.| & matches a number consisting of at least 1 non-zero digit followed by “\verb|00|”, which must be
591 the \emph{last tokens}, because the predicate “\verb|!\.|“ means “must not be followed by a token”
592 \end{tabular}
593
594 \subsection{Choosing between patterns}
595 The character “\verb/|/” between two \<1-patterns> indicates that a match should be made with either the first \emph{or} the second. Obviously, we are not limited to two patterns; there can be as many as desired. If a match occurs for one of the patterns, it is selected, and none of the subsequent patterns are tested: this is therefore an ordered choice. If we write
596 \tktlcode/\r{0-9}^5 | \S{01}^2/
597 the second test will never be performed since it is included in the first. It is therefore important to start with the most specific tests and end with the most general ones if the ranges of the patterns overlap or, worse, are nested.
598
599 The concatenation operator “\verb|:|” takes precedence over the selection operator “\verb/|/”, so
600 \tktlcode/<a> : <b> | <c> | <d> : <e>/
601 is interpreted as \verb|<a> : <b>| or \verb|<c>| or \verb|<d>:<e>|.
602
603 \subsection{Pattern grouping}
604 Any \<1-pattern> or combination of <1-pattern>s denoted by \<patterns> can be enclosed in curly braces to form a new \<1-pattern>, which can, in turn, be preceded by a predicate and followed by a repetition specification according to the syntax described above:
605 \tktlcode/<predicate>{<pattern combination>}<repetitions>/
606
607 \subsection{Spaces}
608 In pattern syntax, spaces are ignored.
609
610 Thus
611 \tktlcode/\r{0-9} ^ 2 : & \r{a-z}^3 : \r{a-z}^ 2/
612 is equivalent to
613 \tktlcode/\r{0-9}^2:&\r{a-z}^3:\r{a-z}^2/
614
615 \subsection{Precedences}
616 The precedences of operators on patterns are (the higher the number, the higher the precedence):
617 \begin{center}
618 \renewcommand\arraystretch{1.25}
619 \begin{tabular}{>\ttfamily p{3.5cm}>\ttfamily p{1.5cm}p{8.5cm}}\hline
620 Operator & \rmfamily Precedence & Description\\\hline
621 \verb|{<patterns>}| & 5 & pattern grouping\\
622 \verb|<1-pattern>?| & 4 & matches 0 or 1 time\\
623 \verb|<1-pattern>*| & 4 & matches 0 or more time\\
624 \verb|<1-pattern>+| & 4 & matches 1 or more times\\
625 \verb|<1-pattern>^{n}| & 4 & matches \verb|n| times\\
626 \verb|<1-pattern>^{a-b}| & 4 & matches between \verb|a| and \verb|b| times\\
627 \verb/!<1-pattern>/ & 3 & matches if \<1-pattern> does not match, without consuming tokens\\
628 \verb/&<1-pattern>/ & 3 & matches if \<1-pattern> matches, without consuming tokens\\
629 \<1-pattern$_1$>\verb/:/\<1-pattern$_2$>/ & 2 & matches \<1-pattern$_1$> then \<1-pattern$_2$>\\
630 \<1-pattern$_1$>\verb/|/\<1-pattern$_2$>/ & 1 & matches \<1-pattern$_1$> or \<1-pattern$_2$> (ordered choice)\\\hline
631 \end{tabular}
632 \end{center}
633
634 \subsection{Pattern names}
635 Any pattern can be defined and named so that it can be reused later using a simpler, easier-to-remember syntax\label{defpattern}:
636 \tktlcode|\defpattern<pattern name>{<patterns>}|
637 The \<pattern name> \emph{must} be a control sequence. This control sequence is not defined or redefined by \verb|\defpattern|; it is also a marker whose meaning is irrelevant.
638
639 \section{Testing for a match with {\ttfamily\textbackslash ifpegmatch}}\label{ifpegmatch}
640 \subsection{Syntax}
641 The \verb|\ifpegmatch| macro has the following syntax
642 \tktlcode/\ifpegmatch[<keys>=<values>]{<patterns>}{<tokens>}{<code if match>}{<code if no match>}/
643
644 When all values are set to their defaults, this macro tests whether the \<patterns> match tokens found at the beginning of the \<tokens>.
645
646 The available \<keys> are:
647 \begin{center}
648 \renewcommand\arraystretch{1.25}
649 \begin{tabular}{>\ttfamily p{3.5cm}>\ttfamily p{3cm}p{8cm}}\hline
650 \rmfamily Key & \rmfamily Default value & Description\\\hline
651 expand arg & 0 & number of expansions the argument containing the \<tokens> must undergo before being considered \\
652 mode & 1 & match search mode \\
653 capture name & \<empty> & capture names \\
654 assign prematch & \verb|\def\prematchtoks| & assignment rule for tokens preceding those that matched \\
655 assign match & \verb|\def\matchtoks| & assignment rule for tokens that have matched \\
656 assign postmatch & \verb|\def\remaintoks| & assignment rule for tokens preceding those that have matched \\\hline
657 \end{tabular}
658 \end{center}
659
660 In this example, we check whether the \<tokens> begin with two digits, an optional space, and at least one lowercase letter
661 \begin{doclatex}
662 \defpattern\okmatch{ \r{0-9:12}^2 : \R{*:10}? : \r{a-z:11}+ }
663 1) \ifpegmatch{\okmatch}{73 ab:*ij}{T}{F}\quad
664 2) \ifpegmatch{\okmatch}{45foobar2000}{T}{F}\quad
665 3) \ifpegmatch{\okmatch}{854tex 8}{T}{F}\quad
666 4) \ifpegmatch{\okmatch}{1 2 b3c}{T}{F}\quad
667 \end{doclatex}
668
669 Let's build a simple grammar to check whether an argument begin with an arithmetic operation of the form \tktlcode/<relative integer><operation><positive integer>/
670 \begin{doclatex}
671 \defpattern\sp{ \R{*:10} } % space
672 \defpattern\digit{ \r{0-9} }% digit
673 \defpattern\posint{ \digit+ }% positive integer
674 \defpattern\int{ \S{+-}? : \posint }% +- integer
675 \defpattern\op{ \S{+-*/} }% math operation
676 \defpattern\okmatch{ \sp* : \int : \sp* : \op : \sp* : \posint : \sp* }
677 1) \ifpegmatch\okmatch{2*3}{T}{F}\quad
678 2) \ifpegmatch\okmatch{-7 + 1 }{T}{F}\quad
679 3) \ifpegmatch\okmatch{ a + 9 }{T}{F}\quad
680 4) \ifpegmatch\okmatch{ -2 / 6 + 3 }{T}{F}\quad
681 5) \ifpegmatch\okmatch{ +2026- 4068}{T}{F}\quad
682 6) \ifpegmatch\okmatch{ -3 }{T}{F}\quad
683 7) \ifpegmatch\okmatch{2a-3b}{T}{F}\par
684 The primitive \verb|\int| is not redefined: $\int x^2dx$
685 \end{doclatex}
686
687 With the predicate “\verb|: !\.|” added to the end of \verb|\okmatch|, line 4 would have displayed “F” because “\Verb*|+ 3 |” remains after the tokens that matched.
688
689 \subsection{Mode}
690 The \verb|\ifpegmatch| macro can search for matches in several modes, specified via the “\verb|mode|” key:
691 \begin{itemize}
692 \item By default, \verb|mode=1| indicates that the match must occur at the beginning of the \<tokens>;
693 \item \verb|mode=0| indicates that all \<tokens> must match; this is the strictest mode;
694 \item \verb|mode=2| indicates that the match can occur anywhere within the \<tokens>; this is the least restrictive mode.
695 \end{itemize}
696
697 \subsection{Token Output}
698 The \verb|\ifpegmatch| macro also returns some information:
699 \begin{itemize}
700 \item by default, the \verb|\matchtoks| macro contains the tokens that matched and is empty if no match was found;
701 \item by default, the \verb|\remaintoks| macro contains the tokens remaining after those that matched;
702 \item by default, the \verb|\prematchtoks| macro contains the tokens preceding those that matched (can only contain tokens when \verb|mode=2|);
703 \item the \verb|\matchposition| macro contains the position of the first token that matched and 0 if no match occurred.
704 \end{itemize}
705
706 \begin{doclatex}
707 1) \ifpegmatch[mode=0]{ \r{A-Z}^3 }{1ABC6}{T}{F}, match=<\matchtoks>, remain=<\remaintoks>, pos=\matchposition\par
708 2) \ifpegmatch[mode=1]{ \r{A-Z}^3 }{1ABC6}{T}{F}, match=<\matchtoks>, remain=<\remaintoks>, pos=\matchposition\par
709 3) \ifpegmatch[mode=2]{ \r{A-Z}^3 }{1ABC6}{T}{F}, match=<\matchtoks>, remain=<\remaintoks>, pos=\matchposition\par
710 4) \ifpegmatch[mode=0]{ \r{A-Z}^3 }{ZZZ12}{T}{F}, match=<\matchtoks>, remain=<\remaintoks>, pos=\matchposition\par
711 5) \ifpegmatch[mode=1]{ \r{A-Z}^3 }{ZZZ12}{T}{F}, match=<\matchtoks>, remain=<\remaintoks>, pos=\matchposition\par
712 6) \ifpegmatch[mode=2]{ \r{A-Z}^3 }{ZZZ12}{T}{F}, match=<\matchtoks>, remain=<\remaintoks>, pos=\matchposition
713 \end{doclatex}
714
715 \subsection{Captures}\label{marqueur:c:capture}
716 The \verb|\c| token, when placed before a \<1-pattern>, indicates that tokens matching this \<1-pattern> must be captured along with their positions. If a \<1-pattern> is a predicate, no capture is performed. Captures are sorted in the chronological order in which they were made and are returned, in two forms, by \verb|\tokscapture{<index>}| or by \verb|\tokscapture{<name>:<index>}|. The \verb|<name>| is optional and is set with the \verb|name| key, and the \verb|<index>| is the capture’s sequence number.
717
718 If \verb|<index>| is 0, all captures are enclosed in curly braces and listed in a CSV file in the following format:
719 \tktlcode/{<capture_1>},{<capture_2>},...,{<capture_n>}/
720
721 Similarly, the macro \verb|\poscapture{<index>}| or \verb|\poscapture{<name>:<index>}| returns the capture positions, with the difference that if \<index> is 0, the positions are placed in a CSV file \emph{without being enclosed in curly braces}:
722 \tktlcode/<position_1>,<position_2>,...,<position_n>/
723
724 If a \<index> exceeds the maximum index, an error message is issued.
725
726 If \verb|\c| appears after a \<1-pattern>, only the position is captured.
727
728 In this example, two complete captures (tokens+position) and one position capture are performed:
729 \begin{doclatex}
730 \ifpegmatch[mode=2]{ \c\r{a-z}+ : \c\r{0-9}^2\c }{12abc666def}{T}{F}\par
731 toks : <\detokenize\expandafter\expandafter\expandafter{\tokscapture{0}}>\qquad
732 1=<\tokscapture{1}>, 2=<\tokscapture{2}>\par
733
734 pos : <\poscapture{0}>\qquad
735 1=<\poscapture{1}>, 2=<\poscapture{2}>, 3=<\poscapture{3}>
736 \end{doclatex}
737
738 In this example, we define a grammar that matches a scientific notation of the form $a\times10^{b}$ and captures both numbers $a$ and $b$:
739 \begin{doclatex}
740 \defpattern\sp{ \R{*:10} }
741 \defpattern\sign{ \S{+-} }
742 \defpattern\digit{ \r{0-9} }
743 \defpattern\integer{ \digit+ }
744 \defpattern\decsep{ \S{.,} }
745 \defpattern\scidec{ \sign? : \r{1-9} : {\decsep : \digit+}? }
746 \defpattern\opbr{ \R{*:1} }
747 \defpattern\clbr{ \R{*:2} }
748 \defpattern\^{ \R{*:7} }
749 \defpattern\exponent{ \opbr : \sp? : \c{\sign? : \sp? : \integer} : \sp? : \clbr | \c\digit }
750 \defpattern\sci{\c\scidec : \sp? : \s{\times10} : \sp? : \^ : \sp? :\exponent}
751 1) \ifpegmatch\sci{3\times10^5} {<\tokscapture{1}> <\tokscapture{2}>}{Faux}\par
752 2) \ifpegmatch\sci{-2.25\times10^{-3}} {<\tokscapture{1}> <\tokscapture{2}>}{Faux}\par
753 3) \ifpegmatch\sci{-0.75\times10^7 } {<\tokscapture{1}> <\tokscapture{2}>}{Faux}\par
754 4) \ifpegmatch\sci{15\times10^0} {<\tokscapture{1}> <\tokscapture{2}>}{Faux}\par
755 5) \ifpegmatch\sci{1.5\times10^ 1 } {<\tokscapture{1}> <\tokscapture{2}>}{Faux}\par
756 6) \ifpegmatch\sci{-2.75 \times 10 ^ { 11 }}{<\tokscapture{1}> <\tokscapture{2}>}{Faux}\par
757 7) \ifpegmatch\sci{-9.96\times10^{ -2 }} {<\tokscapture{1}> <\tokscapture{2}>}{Faux}\par
758 8) \ifpegmatch\sci{-0\times10 ^0} {<\tokscapture{1}> <\tokscapture{2}>}{Faux}\par
759 9) \ifpegmatch\sci{-1\times10^ {- 7 }} {<\tokscapture{1}> <\tokscapture{2}>}{Faux}
760 \end{doclatex}
761
762 \subsection{Recursive Grammars}
763 The \tktl package supports recursive grammars, but no optimization is performed, and the way they are handled remains naive. Consequently, certain recursive grammars—particularly those involving “left recursion”—will result in infinite loops.
764
765 That said, it is possible to use recursive grammars, but adding captures tends to be somewhat unpredictable because the order of these captures is not obvious and depends on the very definition of the grammar and thus on the resulting tree traversal.
766
767 Here is a recursive grammar capturing an arithmetic expression involving the four operations with parentheses:
768 \begin{doclatex}
769 \defpattern\num{ \r{0-9}+ }
770 \defpattern\term{ \num | \s{(} : \expr : \s{)} }
771 \defpattern\factor{ \term : {\S{*/} : \term }* }
772 \defpattern\expr{ \factor : {\S{+-} : \factor }* }
773 1) \ifpegmatch[mode=0]\expr{1+3}{T}{F}\quad
774 2) \ifpegmatch[mode=0]\expr{1-2*3}{T}{F}\quad
775 3) \ifpegmatch[mode=0]\expr{3*4+6}{T}{F}\quad
776 4) \ifpegmatch[mode=0]\expr{3-(1-2*3)}{T}{F}\quad
777 5) \ifpegmatch[mode=0]\expr{3*4*(1-3*2-(1-3/7)*3)/(1/7+2*3)*3-5}{T}{F}\quad
778 6) \ifpegmatch[mode=0]\expr{6-9*(2-3)+4/5}{T}{F}
779 \end{doclatex}
780
781 This grammar ensures that the expression begins with a parenthesis and contains balanced parentheses:
782 \begin{doclatex}
783 \defpattern\nobrtext{ { !\S{()} : \. }+ }
784 \defpattern\inparen{ \nobrtext : \inparen* | \s{(} : \inparen* : \s{)} }
785 \defpattern\expr{ &\s{(} : \inparen : !\. }% predicates -> must start with '(' and end with ')'
786 1) \ifpegmatch\expr{a(b)c}{T}{F}\quad
787 2) \ifpegmatch\expr{(a(abc)()d)}{T}{F}\quad
788 3) \ifpegmatch\expr{(a(bc))df)}{T}{F}\quad
789 4) \ifpegmatch\expr{((abc)d((e)f)g)}{T}{F}\quad
790 5) \ifpegmatch\expr{((foo)b(((b)a)r)}{T}{F}
791 \end{doclatex}
792
793 \section{Counting matches with {\ttfamily\textbackslash pegcount}}\label{pegcount}
794 The macro
795 \tktlcode/\pegcount[<keys>=<values>]{<patterns>}{<tokens>}/
796 counts how many times \<patterns> match in the \<tokens>. Each position is saved, and each match is captured so it can be easily retrieved.
797
798 No capture explicitly requested by \verb|\c| is allowed and is ignored.
799
800 The available \<keys> are:
801 \begin{center}
802 \renewcommand\arraystretch{1.25}
803 \begin{tabular}{>\ttfamily p{3.5cm}>\ttfamily p{2.5cm}p{8cm}}\hline
804 \rmfamily Key & \rmfamily Default value & Description\\\hline
805 expand arg & 0 & number of expansions the argument containing the \<tokens> must undergo before being taken into account.\\
806 assign & \<empty> & if empty, displays the number of matches. Otherwise, must contain an assignment statement of the type
807 \verb|\def<macro>| to assign the result to the \<macro>\\
808 assign positions & \verb|\def\matchposlist| & assignment statement for the list of positions. If \<empty>, no capture is performed. Otherwise, must
809 contain an assignment statement \verb|\def<macro>| to assign the list of positions to a \<macro>\\
810 name & \<empty> & is the name of the captures returned by \verb|\tokscapture|\\\hline
811 \end{tabular}
812 \end{center}
813
814 Captures are returned, in two expansions, by \verb|\tokscapture{[<name>:]<index>}|. The \verb|<name>| is optional and is set using the \verb|<name>| key, and the \verb|<index>| is the capture index. If \verb|<index>| is 0, all captures are enclosed in curly braces and listed in a CSV file in the following format:
815 \tktlcode/{<capture_1>},{<capture_2>},...,{<capture_n>}/
816
817 \begin{doclatex}
818 1) \pegcount{ \r{0-9}+ : \r{a-z}+ }{foo25bar}, <\matchposlist>\par% numbers followed by letters
819 2) \pegcount{ \r{0-9}+ : \r{a-z}+ }{a12bcd,4b,z875bar}, <\matchposlist>,
820 "\tokscapture{0}", 1="\tokscapture{1}", 2="\tokscapture{2}", 3="\tokscapture{3}"\par% numbers followed by letters
821 3) \pegcount{ \s{+} : {\r{a-c}^2 : \r{0-9}}+ : \s{+} }{+ab3+..+bb6ab8ca7+..+aa1bb2+..},
822 <\matchposlist>,
823 1="\tokscapture{1}", 2="\tokscapture{2}", 3="\tokscapture{3}"
824 \end{doclatex}
825
826 \section{Replacements with {\ttfamily\textbackslash pegreplace}}\label{pegreplace}
827 The macro
828 \tktlcode/\pegreplace[<keys>=<values>]
829 {
830 <patterns_1> -&> <remplacment_1>,
831 <patterns_2> -&> <remplacment_2>,
832 etc.
833 <patterns_n> -&> <remplacment_n>
834 }{<tokens>}/
835 searches for all \<patterns> within the \<tokens> and, for each match, replaces the tokens that matched with the code defined in \<replacement>.
836
837 The available \<keys> are:
838 \begin{center}
839 \renewcommand\arraystretch{1.25}
840 \begin{tabular}{>\ttfamily p{3.5cm}>\ttfamily p{2.5cm}p{8cm}}\hline
841 \rmfamily Key & \rmfamily Default value & Description\\\hline
842 expand arg & 0 & number of expansions the argument containing the \<tokens> must undergo before being taken into account.\\
843 mode & 2 & specifies the mode in which \verb|\pegreplace| should search for matches\\
844 assign & \<empty> & if empty, displays the \<tokens> obtained after performing the replacements. Otherwise, must contain an
845 assignment statement to assign the result to a macro using \verb|\def<macro>| or to a token register.\\\hline
846 \end{tabular}
847 \end{center}
848
849 It is important to understand that all positions in \<tokens> are considered, from position $1$ to position $L$, where $L$ is the number of tokens. For each position, all \<patterns> are tested in turn to determine whether a match occurs at that position. If a \<pattern$_i$> matches at position $k$, the corresponding \<replacement$_i$> is performed, and \verb|\pegreplace| is ready to move to position $k+1$. The key \verb|mode| must be an integer between 0 and 2 that specifies how the \verb|\pegreplace| macro should behave after the first match has been found and the first replacement made:
850 \begin{itemize}
851 \item \verb|mode=0| indicates that after the first match at position $k$, no other positions will be examined, no further actions will be performed, and the \verb|\pegreplace| macro has finished its work;
852 \item \verb|mode=1| indicates that all subsequent positions will be examined, but if a \<pattern$_k$> matches, it is then neutralized and cannot match again; in other words, each \<pattern> can match at most once;
853 \item \verb|mode=2|, which is the default value, indicates that all subsequent positions will be examined and that each \<pattern> can match as many times as necessary.
854 \end{itemize}
855
856 The \<replacement> is an arbitrary code that does not contain a comma; it may include “\verb|\0|”\label{maqueur:0}, which means “tokens that matched,” “\verb|\1|” is the first capture made by \verb|\c|, “\verb|\2|” is the second, and so on up to “\verb|\9|.” Note that spaces before and after \<replacement> are removed. If a \<replacement> is likely to contain a comma or if you want to preserve the spaces before or after it, you must enclose the entire expression in curly braces.
857
858 For example, to convert each word into a CSV list and add spaces, you would need to write
859 \begin{doclatex}
860 \pegreplace{
861 \r{a-z,A-Z} : &\. : !\R{*:10} -> {\0, } ,% braces necessary here
862 \R{*:10} -> \quad
863 }{Happy TeXing}
864 \end{doclatex}
865
866 To show how \verb|\pegreplace| behaves depending on the value of the \verb|mode| key, in this example, we enclose any sequence of two lowercase letters and place any number from 1 to 5 between angle brackets:
867 \begin{doclatex}
868 \fboxsep=1pt
869 \pegreplace[mode=0]{
870 \r{a-z}^2 -> \fbox{\strut\0},
871 \S{12345} -> $\langle\0\rangle$
872 }{6foob1baz327z}
873
874 \pegreplace[mode=1]{
875 \r{a-z}^2 -> \fbox{\strut\0},
876 \S{12345} -> $\langle\0\rangle$
877 }{6foob1baz327z}
878
879 \pegreplace[mode=2]{
880 \r{a-z}^2 -> \fbox{\strut\0},
881 \S{12345} -> $\langle\0\rangle$
882 }{6foob1baz327z}
883 \end{doclatex}
884
885 Framing of \verb|\alpha| and \verb|\beta| with their coefficients in math mode:
886 \begin{doclatex}
887 \fboxsep=1pt
888 \pegreplace{ \r{1-9}* : \S{\alpha\beta} -> \fbox{\strut$\0$} }{$2\alpha-3\beta=-4-\alpha+4\beta$}
889 \end{doclatex}
890
891 In this example, we create a grammar that matches the French standards for postal codes and city names (consisting of words and hyphens).
892 \begin{doclatex}
893 \defpattern\sp{ \R{*:10} }
894 \defpattern\CP{\c\r{0-9}^2 : \sp? : \c\r{0-9}^3 }% \1=2 first digits \2=3 last digits
895 \defpattern\upcase{ \r{A-Z,À,É} }
896 \defpattern\lowcase{ \r{a-z,é,è,à,ê,ô,ç} }
897 \defpattern\ville{ \upcase : \lowcase+ : { \S{-} : {\upcase | \lowcase} : \lowcase+ }* }
898 \defpattern\CPville{ \CP : \sp : \c\ville }% capture CP and capture name
899 1) \pegreplace{\CPville -> CodePostal=\textbf{\1\2} est \fbox{\3} }{Destination 75000 Paris}\par
900 2) \pegreplace{\CPville -> CodePostal=\textbf{\1\2} est \fbox{\3} }{Destination 64 500 Saint-Jean-de-Luz suite}\par
901 3) \pegreplace{\CPville -> CodePostal=\textbf{\1\2} est \fbox{\3} }{Destination 38120 Saint-Égrève}
902 \end{doclatex}
903
904 Let's come up with a little riddle involving two-digit numbers that are not zero:
905 \begin{doclatex}
906 \defpattern\num{ \c\r{1-9} : \c\r{1-9} : !\r{0-9} }
907 If \pegreplace{\num -> \1\2 gives \the\numexpr\1*\2+\2\relax }{27, 34 and 43}, what gives 57 ? (Answer in 7.5 million years)
908 \end{doclatex}
909
910 \section{Errors}
911 Among the many errors that can occur, here are a few...
912
913 \subsection{Unbalanced Braces}
914 If a result or capture consists of tokens with unbalanced braces, a compilation error will occur.
915
916 For example, if you use \verb|\toksdo| to remove the closing braces:
917 \tktlcode/\toksdo{ \R{*:2} -&> \deltok }{foo{\bfseries123}bar}/
918 the error reported by \tktl is “\texttt{! Unbalanced open-group token, 1 close-group token added}”.
919
920 To balance the braces, a closing brace is therefore added to the end of the tokens passed as arguments, and the tokens “\verb|foo{\bfseries123bar}|” are sent to the display.
921
922 Similarly, if we remove the opening braces:
923 \tktlcode/\toksdo{ \R{*:1} -&> \deltok}{foo{\bfseries123}bar}/
924 the error message is “\texttt{! Unbalanced close-group token ignored}”.
925
926 The extra closing brace is therefore ignored, and the tokens “\verb|foo\bfseries123bar|” are sent to the display
927
928 \subsection{Engine not suitable for encoding}
929 Using an 8-bit engine \emph{requires} the use of tokens only, particularly for the \verb|\r| token.
930
931 Compiling this code with an 8-bit engine
932 \tktlcode/\ifpegmatch{ \r{a,e,i,o,u,y,é,à,ê,ù} }{Un été à l'océan}{T}{F}/
933 causes a compilation error because “\verb|é|” consists of 2 tokens (charcodes 195 and 169, catcodes equal to 13), whereas the syntax of \verb|\r| requires that there be only one. The compilation error reported by \tktl is “\texttt{! Multiple token 'é', '0' inserted}”.
934
935 The correct syntax would be
936 \tktlcode/\ifpegmatch{ \r{a,e,i,o,u,y} | \S{éàêù} }{Un été à l'océan}{T}{F}/
937
938 \subsection{Invalid Interval}
939 Intervals of the form \verb|<a>-<b>|, where \verb|a| and \verb|b| are either tokens or numbers, \emph{must} be in the correct order.
940
941 Since the token “ń” comes \emph{after} “ó” in UTF-8, the code
942 \tktlcode/\ifpegmatch{ \r{ń-ó} }{Some text}{T}{F}/
943 causes the compilation error ”\texttt{! Unsorted interval in 'ń-ó', 'ó-ń' inserted}”. The range is corrected by \tktl.
944
945 \subsection{Invalid Pattern Syntax}
946 Spaces are ignored, but any pattern syntax that does not conform to what is described starting on page~\pageref{patterns} will cause a compilation error.
947
948 For example
949 \tktlcode/\pegcount{ \r{a-z} | S{10} }{ab1023truc098}/
950 causes the error ”\texttt{! Found "S" when expecting \string\r, \string\R, \string\s, \string\S\space or \string\.}”.
951
952 \subsection{Capture Errors}
953 If a \<name> has not been defined or if a \<index> is requested outside of those assigned during captures, a compilation error is returned.
954
955 \tktlcode/\pegcount{ \r{a-z}^2 }{ab1023truc098}/
956 creates 3 captures, so requesting
957 \tktlcode/\tokscapture{4}/
958 results in the compilation error “\texttt{! Undefined token capture at index "4"}”.
959
960 \section{List of macros and markers}
961 \subsection{List of commands}
962 Here are the macros available to the user:
963 \begin{itemize}
964 \item \verb|\printtoks|, see page~\pageref{printtoks};
965 \item \verb|\toksdo|, \verb|\setcharcode|, \verb|\setcatcode|, \verb|\deltok|, \verb|\addtok|, \verb|\selfcharcode|, \verb|\selfcatcode| and \verb|\selfindex|, see pages~\pageref{toksdo} and following;
966 \item \verb|\tokscount|, see page~\pageref{tokscount};
967 \item \verb|\defpattern|, see page~\pageref{defpattern};
968 \item \verb|\ifpegmatch|, see page~\pageref{ifpegmatch};
969 \item \verb|\pegcount|, see page~\pageref{pegcount};
970 \item \verb|\pegreplace|, see page~\pageref{pegreplace}.
971 \end{itemize}
972 The following macros are modified by the \verb|\toksdo| macro, but are restored to their previous state once \verb|\toksdo| has finished executing:
973 \verb|\setcharcode|, \verb|\setcatcode|, \verb|\deltok|, \verb|\addtok|, \verb|\selfcharcode|, \verb|\selfcatcode| and \verb|\selfindex|.
974
975 \subsection{Liste of markers}
976 \begin{itemize}[topsep=0pt]
977 \item\verb|\r{<csv car>:<csv catcode>}|, see page~\pageref{motif:r};
978 \item\verb|\R{<csv charcodes>:<csv catcode>}|, see page~\pageref{motif:R};
979 \item\verb|\s{<tokens>}|, see page~\pageref{motif:s};
980 \item\verb|\S{<tokens>}|, see page~\pageref{motif:S};
981 \item\verb|\.|, see page~\pageref{motif:dot};
982 \item\verb|\c| has two syntaxes
983 \begin{itemize}[label={}]
984 \item \verb|\c{<catcode>}{<tokens>}| when used in the argument of \verb|\s| or \verb|\S|, see page~\pageref{marqueur:c:catcode};
985 \item \verb|\c| when placed before or after a \<1-pattern>, see page~\pageref{marqueur:c:capture};
986 \end{itemize}
987 \item\verb|\self|, see page~\pageref{marqueur:self};
988 \item \verb|\0|, \verb|\1| up to \verb|\9|, see page~\pageref{marqueur:0};
989 \item any \<macro>, whether it exists or not, passed as the first argument to \verb|\defpattern|.
990 \end{itemize}
991
992 \begin{center}
993 \parskip0pt
994 $\star$\par
995 $\star\quad\star$
996 \end{center}
997
998 This package, version \tktlver{}, is still in the experimental stage, and it is possible that, despite the extensive testing that has been done, it contains many bugs.
999
1000 In addition, some features or syntax are still likely to be slightly modified.
1001
1002 Anyway, I hope \tktl is useful to you. Feel free to contact me by \href{mailto:[email protected]}{\texttt{\textbf{email}}} to report any bugs, malfunctions, or suggestions for realistic features. Above all, please do not waste your time posting them on \verb|https://tex.stackexchange.com| or any other site, as there is a very good chance I won't see them.
1003 \end{document}