| 1 |
\iffalse meta-comment |
| 2 |
|
| 3 |
File: l3term-glossary.tex |
| 4 |
|
| 5 |
Copyright (C) 2018-2026 The LaTeX Project |
| 6 |
|
| 7 |
It may be distributed and/or modified under the conditions of the |
| 8 |
LaTeX Project Public License (LPPL), either version 1.3c of this |
| 9 |
license or (at your option) any later version. The latest version |
| 10 |
of this license is in the file |
| 11 |
|
| 12 |
https://www.latex-project.org/lppl.txt |
| 13 |
|
| 14 |
This file is part of the "l3kernel bundle" (The Work in LPPL) |
| 15 |
and all files in that bundle must be distributed together. |
| 16 |
|
| 17 |
The released version of this bundle is available from CTAN. |
| 18 |
|
| 19 |
\fi |
| 20 |
|
| 21 |
\documentclass{l3doc} |
| 22 |
|
| 23 |
|
| 24 |
\title{% |
| 25 |
Glossary of \TeX{} terms used to describe \LaTeX3 functions% |
| 26 |
} |
| 27 |
\author{% |
| 28 |
The \LaTeX{} Project\thanks |
| 29 |
{% |
| 30 |
E-mail: |
| 31 |
\href{mailto:[email protected]}% |
| 32 |
{[email protected]}% |
| 33 |
}% |
| 34 |
} |
| 35 |
\date{Released 2026-05-15} |
| 36 |
|
| 37 |
\newcommand{\TF}{\textit{(TF)}} |
| 38 |
|
| 39 |
\begin{document} |
| 40 |
|
| 41 |
\maketitle |
| 42 |
|
| 43 |
This file describes aspects of \TeX{} programming that are relevant in a |
| 44 |
\pkg{expl3} context. |
| 45 |
|
| 46 |
\section{Reading a file} |
| 47 |
|
| 48 |
Tokenization. |
| 49 |
|
| 50 |
Treatment of spaces, such as the trap that \verb|\~~a| is equivalent to |
| 51 |
\verb|\~a| in \pkg{expl3} syntax, or that \verb|~| fails to give a space at the |
| 52 |
beginning of a line. |
| 53 |
|
| 54 |
\section{Structure of tokens} |
| 55 |
|
| 56 |
We refer to the documentation of \texttt{l3token} for a complete |
| 57 |
description of all \TeX{} tokens. We distinguish the meaning of the |
| 58 |
token, which controls the expansion of the token and its effect on |
| 59 |
\TeX{}'s state, and its shape, which is used when comparing token lists |
| 60 |
such as for delimited arguments. At any given time two tokens of the |
| 61 |
same shape automatically have the same meaning, but the converse does |
| 62 |
not hold, and the meaning associated with a given shape change when |
| 63 |
doing assignments. |
| 64 |
|
| 65 |
Apart from a few exceptions, a token has one of the following shapes. |
| 66 |
\begin{itemize} |
| 67 |
\item A control sequence, characterized by the sequence of characters |
| 68 |
that constitute its name: for instance, \cs{use:n} is a five-letter |
| 69 |
control sequence. |
| 70 |
\item An active character token, characterized by its character code |
| 71 |
(between $0$ and $1114111$ for \LuaTeX{} and \XeTeX{} and less for |
| 72 |
other engines) and category code~$13$. |
| 73 |
\item A character token such as |A| or |#|, characterized by its |
| 74 |
character code and category code (one of $1$, $2$, $3$, $4$, $6$, |
| 75 |
$7$, $8$, $10$, $11$ or~$12$ whose meaning is described below). |
| 76 |
\end{itemize} |
| 77 |
|
| 78 |
The meaning of a (non-active) character token is fixed by its category |
| 79 |
code (and character code) and cannot be changed. We call these tokens |
| 80 |
\emph{explicit} character tokens. Category codes that a character token |
| 81 |
can have are listed below by giving a sample output of the \TeX{} |
| 82 |
primitive \tn{meaning}, together with their \pkg{expl3} names and most |
| 83 |
common example: |
| 84 |
\begin{itemize} |
| 85 |
\item[1] begin-group character (|group_begin|, often |{|), |
| 86 |
\item[2] end-group character (|group_end|, often |}|), |
| 87 |
\item[3] math shift character (|math_toggle|, often |$|), % $ |
| 88 |
\item[4] alignment tab character (|alignment|, often |&|), |
| 89 |
\item[6] macro parameter character (|parameter|, often |#|), |
| 90 |
\item[7] superscript character (|math_superscript|, often |^|), |
| 91 |
\item[8] subscript character (|math_subscript|, often |_|), |
| 92 |
\item[10] blank space (|space|, often character code~$32$), |
| 93 |
\item[11] the letter (|letter|, such as |A|), |
| 94 |
\item[12] the character (|other|, such as |0|). |
| 95 |
\end{itemize} |
| 96 |
Category code~$13$ (|active|) is discussed below. Input characters can |
| 97 |
also have several other category codes which do not lead to character |
| 98 |
tokens for later processing: $0$~(|escape|), $5$~(|end_line|), |
| 99 |
$9$~(|ignore|), $14$~(|comment|), and $15$~(|invalid|). |
| 100 |
|
| 101 |
The meaning of a control sequence or active character can be identical |
| 102 |
to that of any character token listed above (with any character code), |
| 103 |
and we call such tokens \emph{implicit} character tokens. The meaning |
| 104 |
is otherwise in the following list: |
| 105 |
\begin{itemize} |
| 106 |
\item a macro, used in \pkg{expl3} for most functions and some variables |
| 107 |
(|tl|, |fp|, |seq|, \ldots{}), |
| 108 |
\item a primitive such as \tn{def} or \tn{topmark}, used in \pkg{expl3} |
| 109 |
for some functions, |
| 110 |
\item a register such as \tn{count}|123|, used in \pkg{expl3} for the |
| 111 |
implementation of some variables (|int|, |dim|, \ldots{}), |
| 112 |
\item a constant integer such as \tn{char}|"56| or |
| 113 |
\tn{mathchar}|"121|, used when defining a constant using |
| 114 |
\cs{int_const:Nn}, |
| 115 |
\item a font selection command, |
| 116 |
\item undefined. |
| 117 |
\end{itemize} |
| 118 |
Macros can be \tn{protected} or not, \tn{long} or not (the opposite of |
| 119 |
what \pkg{expl3} calls |nopar|), and \tn{outer} or not (unused in \pkg{expl3}). |
| 120 |
Their \tn{meaning} takes the form |
| 121 |
\begin{quote} |
| 122 |
\meta{prefix} |macro:|\meta{argument}|->|\meta{replacement} |
| 123 |
\end{quote} |
| 124 |
where \meta{prefix} is among \tn{protected}\tn{long}\tn{outer}, |
| 125 |
\meta{argument} describes parameters that the macro expects, such as |
| 126 |
|#1#2#3|, and \meta{replacement} describes how the parameters are |
| 127 |
manipulated, such as~|\int_eval:n{#2+#1*#3}|. This information can be |
| 128 |
accessed by \cs{cs_prefix_spec:N}, \cs{cs_parameter_spec:N}, |
| 129 |
\cs{cs_replacement_spec:N}. |
| 130 |
|
| 131 |
When a macro takes an undelimited argument, explicit space characters |
| 132 |
(with character code $32$ and category code $10$) are ignored. If the |
| 133 |
following token is an explicit character token with category code $1$ |
| 134 |
(begin-group) and an arbitrary character code, then \TeX{} scans ahead |
| 135 |
to obtain an equal number of explicit character tokens with category |
| 136 |
code $1$ (begin-group) and $2$ (end-group), and the resulting list of |
| 137 |
tokens (with outer braces removed) becomes the argument. Otherwise, a |
| 138 |
single token is taken as the argument for the macro: we call such single |
| 139 |
tokens \enquote{N-type}, as they are suitable to be used as an argument |
| 140 |
for a function with the signature~\texttt{:N}. |
| 141 |
|
| 142 |
When a macro takes a delimited argument \TeX{} scans ahead until finding |
| 143 |
the delimiter (outside any pairs of begin-group/end-group explicit |
| 144 |
characters), and the resulting list of tokens (with outer braces |
| 145 |
removed) becomes the argument. Note that explicit space characters at |
| 146 |
the start of the argument are \emph{not} ignored in this case (and they |
| 147 |
prevent brace-stripping). |
| 148 |
|
| 149 |
\section{Handling of hash tokens} |
| 150 |
|
| 151 |
\TeX{} uses the hash (octothorpe) character |#| to denote parameters for |
| 152 |
macros: these must be numbered sequentially. To allow handling of nested |
| 153 |
macros, \TeX{} requires that for each nesting level, hash tokens are doubled. |
| 154 |
For example |
| 155 |
\begin{verbatim} |
| 156 |
\cs_new:Npn \mypkg_outer:N #1 |
| 157 |
{ |
| 158 |
\cs_new:Npn \mypkg_inner:N ##1 |
| 159 |
{ |
| 160 |
#1 |
| 161 |
##1 |
| 162 |
} |
| 163 |
} |
| 164 |
\end{verbatim} |
| 165 |
would define both |\mypkg_outer:N| and |\mypkg_inner:N| as taking |
| 166 |
exactly one argument. If we then do |
| 167 |
\begin{verbatim} |
| 168 |
\mypkg_outer:N \foo |
| 169 |
\cs_show:N \mypkg_inner:N |
| 170 |
\end{verbatim} |
| 171 |
\TeX{} will report |
| 172 |
\begin{verbatim} |
| 173 |
> \mypkg_inner:N=\long macro:#1->\foo #1. |
| 174 |
\end{verbatim} |
| 175 |
i.e., the hash is not doubled, but is now the parameter of this macro. |
| 176 |
|
| 177 |
Exactly the same concept applies to anywhere that inline code is nested in |
| 178 |
\pkg{expl3}, for example inline mapping code, key definitions, etc. |
| 179 |
|
| 180 |
\end{document} |