Debian Codegraph @

texmf-dist/doc/generic/memoize/memoize-doc.tex

raw
1 \documentclass[a4paper,11pt]{article}
2 \usepackage[margin=2cm,top=1cm,bottom=1cm,nohead,includefoot]{geometry}
3
4 % The perl-based extraction chokes on some externs.
5 \usepackage[extract=python]{nomemoize}
6 \usepackage{memoize-doc}
7 \mmzset{
8 % recompile,
9 % readonly,
10 % disable,
11 % inverse search=false,
12 % include context in ccmemo,
13 % direct ccmemo input,
14 % trace,
15 }
16 \def\exampledir{examples/}
17 \def\attachmentsdir{examples/attachments/}
18
19 \title{\pkg[white]{Memoize}}
20 \ParseProvidesPackage
21 \ProvidesPackage{memoize}[2024/12/02 v1.4.1 Fast and flexible externalization]
22 \packageversion{v\packagever\\\csuse{@date}}
23 \author{Sašo Živanović\\[2mm]
24 \emailsymbol~\url(mailto:){[email protected]}\\
25 \homepagesymbol~\url(http://){spj.ff.uni-lj.si/zivanovic}\\
26 \faGithub~\url(https://){github.com/sasozivanovic}}
27
28 \hypersetup{
29 pdftitle={Memoize},
30 pdfauthor={Sašo Živanović},
31 pdfsubject={externalization},
32 pdfkeywords={LaTeX, externalization, memoization}
33 }
34
35 \usepackage{makeidx}
36 % \pdfsystem{makeindex -s memoize-doc.mst \jobname.idx}
37 \makeindex
38 \usepackage[hangindent=1em,justific=raggedright,font=small,totoc]{idxlayout}
39 \appto\indexfont{\def\pkgcolor{gray}}
40
41 \AddToHook{begindocument}{% so that the region compilation takes these in
42 \docForeign{
43 scope={
44 after/.style={index annotation/.prefix={\hologo{LaTeX}\space}},
45 cmd={name=begin}, cmd={name=end},
46 cmd={name=label}, cmd={name=@currentlabel},
47 cmd={name=ref}, cmd={name=pageref},
48 cmd={name=index},
49 cmd={name=NewDocumentCommand}, cmd={name=GetDocumentCommandArgSpec},
50 env={name=verbatim},
51 cmd={name=ReadonlyShipoutCounter},
52 hook={name=begindocument},
53 cmd={name=usepackage},
54 },
55 scope={
56 before/.style={index annotation=\hologo{TeX} primitive},
57 cmd={name=shipout},
58 cmd={name=hbox}, cmd={name=vbox},
59 cmd={name=ignorespaces}, cmd={name=unskip},
60 cmd={name=errmessage},
61 cmd={name=catcode},
62 cmd={name=mag},
63 },
64 scope={
65 before/.style={index annotation=\hologo{eTeX} primitive},
66 cmd={name=scantokens},
67 cmd={name=currentgrouplevel},
68 },
69 scope={
70 before/.style={index annotation=\hologo{pdfTeX} primitive},
71 cmd={name=quitvmode},
72 },
73 scope={
74 before/.style={index annotation=\hologo{LuaTeX} primitive},
75 pdfcmd={name=savepos},
76 cmd={name=pdfvariable},
77 },
78 pdfvariable={name=compresslevel},
79 pdfvariable={name=horigin},
80 pdfvariable={name=vorigin},
81 scope={
82 after/.style={index annotation/.prefix={\hologo{ConTeXt}\space}},
83 cmd={name=realpageno},
84 },
85 pdfcmd={name=primitive, index annotation=\hologo{LuaTeX}/\hologo{XeTeX} primitive},
86 register={name=pdfmajorversion},
87 register={name=pdfminorversion},
88 scope={
89 after/.style={index annotation/.prefix={\hologo{plainTeX}\space}},
90 cmd={name=llap}, cmd={name=rlap},
91 },
92 scope={
93 pkg=forest,
94 env={name=forest},
95 cmd={name=Forest},
96 },
97 scope={
98 pkg=tikz, keypath=/tikz,
99 cmd={name=tikz},
100 env={name=tikzpicture},
101 key={name=remember picture},
102 key={name=overlay},
103 cmd={name=tikzexternalize},
104 cmd={name=pgfsys@getposition},
105 },
106 scope={
107 pkg=beamer,
108 env={name=frame},
109 },
110 file={name=texmf.cnf},
111 env={name=tcblisting, pkg=tcolorbox},
112 scope={
113 keypath=/tcb, pkg=tcolorbox,
114 key={name=breakable},
115 key={name=float},
116 },
117 cmd={name=vref, pkg=varioref},
118 cmd={name=cref, pkg=cleveref},
119 cmd={name=crefrange, pkg=cleveref},
120 xparse type={name=l, index annotation/.append={\ (up to begin-group)}},
121 xparse type={name=u, index annotation/.append={\ (until tokens)}},
122 xparse type={name=r, index annotation/.append={\ (required delimited)}},
123 xparse type={name=R, index annotation/.append={\ (required delimited with default)}},
124 xparse type={name=d, index annotation/.append={\ (optional delimited)}},
125 xparse type={name=D, index annotation/.append={\ (optional delimited with default)}},
126 xparse type={name=o, index annotation/.append={\ (standard optional)}},
127 xparse type={name=O, index annotation/.append={\ (standard optional with default)}},
128 xparse type={name=t, index annotation/.append={\ (optional token)}},
129 xparse type={name=s, index annotation/.append={\ (optional star)}},
130 xparse type={name=m, index annotation/.append={\ (mandatory)}},
131 xparse type={name=g, index annotation/.append={\ (optional group)}},
132 xparse type={name=G, index annotation/.append={\ (optional group with default)}},
133 xparse type={name=v, index annotation/.append={\ (verbatim)}},
134 xparse type={name=e, index annotation/.append={\ (embellishments)}},
135 xparse type={name=E, index annotation/.append={\ (embellishments with defaults)}},
136 xparse modifier={name=+, index annotation/.append={\ (long)}},
137 xparse modifier={name=!, index annotation/.append={\ (don't ignore spaces)}},
138 xparse modifier={name=>, index annotation/.append={\ (argument processor)}},
139 generic={name=shell-escape, name prefix=-, label prefix:=-,
140 index annotation=option of \hologo{TeX} binaries},
141 generic={name=enable-write18, name prefix=--, label prefix:=--,
142 index annotation=option of \hologo{TeX} binaries},
143 generic={name=TEXMFOUTPUT, index annotation={environment variable,
144 or variable in \reffile[into index=false]{texmf.cnf}}},
145 generic={name=TEXMF\_OUTPUT\_DIRECTORY, label=TEXMF_OUTPUT_DIRECTORY,
146 index annotation={environment variable, or variable in
147 \reffile[into index=false]{texmf.cnf}}},
148 generic={name=TEXLIVE\_WINDOWS\_TRY\_EXTERNAL\_PERL,
149 label=TEXLIVE_WINDOWS_TRY_EXTERNAL_PERL,
150 index annotation={variable in \reffile[into index=false]{texmf.cnf}}},
151 generic={name=openout\_any, label=openout_any,
152 index annotation={variable in \reffile[into index=false]{texmf.cnf}}},
153 generic={name=openin\_any, label=openin_any,
154 index annotation={variable in \reffile[into index=false]{texmf.cnf}}},
155 file={pkg=latexmk, name=memoize\_latexmkrc, label=memoize_latexmkrc},
156 }%
157 \docindex{%
158 cmd={name=pdfprimitive, see={\refcmd[into index=false,short]{primitive}}},
159 cmd={name=pdfsavepos, see={\refcmd[into index=false,short]{savepos}}},
160 cmd={name=pdfhorigin, see={\docref[into index=false]{pdfvar:horigin}}},
161 cmd={name=pdfvorigin, see={\docref[into index=false]{pdfvar:vorigin}}},
162 cmd={name=pdfcompresslevel, see={\docref[into index=false]{pdfvar:compresslevel}}},
163 pdfvariable={name=majorversion, see={\docref[into index=false,long]{reg:pdfmajorversion}}},
164 pdfvariable={name=minorversion, see={\docref[into index=false,long]{reg:pdfminorversion}}},
165 }%
166 }
167
168 \begin{document}
169
170
171 \maketitle
172 \thispagestyle{empty}
173
174 \begin{abstract}
175 Memoize is a package for externalization of graphics and memoization of
176 compilation results in general, allowing the author to reuse the results of
177 compilation-intensive code.
178 Memoize
179 \begin{enumerate*}[(i)]
180 \item induces very little overhead, as all externalized graphics is produced
181 in a single compilation. It features
182 \item automatic recompilation upon the change of code or user-adjustable
183 context, and
184 \item automatic externalization of \TikZ pictures and Forest trees, easily
185 extensible to other commands and environments. Furthermore, Memoize
186 \item supports cross-referencing, \TikZ overlays and Beamer,
187 \item works with all major engines and formats, and
188 \item is adaptable to any workflow.
189 \end{enumerate*}
190 \end{abstract}
191
192 \vfill
193
194 \begingroup
195 \tcbset{page box/.style={
196 box align=center,boxsep=2mm,top=0pt,bottom=0pt,left=0pt,right=0pt,
197 enhanced, remember as=#1,
198 }}%
199 \begin{tcboxedraster}
200 [raster columns=1]
201 {title=The two steps of externalization of graphics in Memoize}
202 \tcbinputexample{titlepage}{
203 title=\tt doc.tex,
204 listing only,
205 remember as=tex,
206 }
207 \begin{tcolorbox}[example title=doc.pdf, no attachment, remember as=pdf1]
208 \includeexamplepdf[page box=extern1][titlepage](.c1){page=1,trim=1in 1in 1in 1in}\hfill
209 \includeexamplepdf[page box=extern2][titlepage](.c1){page=2,trim=1in 1in 1in 1in}\hfill
210 \includeexamplepdf[page box=extern3][titlepage](.c1){page=3,trim=1in 1in 1in 1in}\hfill
211 \includeexamplepdf[page box=docpage][titlepage](.c1){page=4}%
212 \end{tcolorbox}
213 \vspace{10mm}
214 \begin{tcboxedraster}[raster columns=3, raster column skip=1cm]{blankest}
215 \begin{tcolorbox}[example title=.pdf]\centering
216 \includeexamplepdf[page box=externfile1][titlepage](.c1){page=1,trim=1in 1in 1in 1in}\hfill
217 \end{tcolorbox}
218 \begin{tcolorbox}[example title=.pdf]\centering
219 \includeexamplepdf[page box=externfile2][titlepage](.c1){page=2,trim=1in 1in 1in 1in}\hfill
220 \end{tcolorbox}
221 \begin{tcolorbox}[example title=.pdf]\centering
222 \includeexamplepdf[page box=externfile3][titlepage](.c1){page=3,trim=1in 1in 1in 1in}\hfill
223 \end{tcolorbox}
224 \end{tcboxedraster}
225 \end{tcboxedraster}
226
227 \begin{tcolorbox}[title=Using the externalized graphics]
228 \begin{tcolorbox}[example title=doc.pdf,
229 halign=center, remember as=pdf2]
230 \tcbox[remember, enhanced]{\footnotesize
231 We all love Ti\emph{k}Zlings!
232 \def\mytikzling#1#2{%
233 \tcbox[blankest, baseline=1mm, remember as=used extern #1]
234 {\tikz[x=0.3cm,y=0.3cm]{#2}}}%
235 \mytikzling1{\penguin} is a penguin,
236 \mytikzling2{\koala} is a koala, and
237 \mytikzling3{\owl} is an owl.
238 }
239 \end{tcolorbox}
240 \end{tcolorbox}
241
242 \vfill
243
244 \begin{tcolorbox}[colback=emphcolor, fontupper=\footnotesize]
245 This manual also documents packages Advice
246 ({\ParseProvidesPackage
247 \ProvidesPackage{advice}[2024/03/15 v1.1.1 Extend commands and environments]
248 v\packagever}) and CollArgs
249 ({\ParseProvidesPackage
250 \ProvidesPackage{collargs}[2024/03/15 v1.2.0 Collect arguments of any command]
251 v\packagever}). These are auxiliary
252 packages which were developed alongside Memoize, but are distributed as
253 independent packages as they might be useful outside the context of Memoize,
254 as well. See sections~\ref{sec:advice} and \ref{sec:ref:advice} for Advice, and
255 sections~\ref{sec:collargs} and \ref{sec:ref:collargs} for CollArgs.
256 \begin{tikzpicture}[remember picture, overlay]
257 \draw[->, snake arrow, thick, red] (tex.west) to[out=-90-60, in=90+60] (pdf1.west);
258 \draw[->, snake arrow, thick, red] (tex.east) to[out=-90+19, in=90-19] (pdf2.east);
259 \draw[->, thick, red] (extern1.south) to[out=south, in=north] (externfile1);
260 \draw[->, thick, red] (extern2.south) to[out=south, out looseness=1.1, in=north, in looseness=0.9] (externfile2);
261 \draw[->, thick, red] (extern3.south) to[out=south, out looseness=0.7, in=north, in looseness=0.9] (externfile3);
262 \draw[->, thick,blue] (externfile1) to[out=south, in=north] ([yshift=0.8mm]used extern 1.north);
263 \draw[->, thick,blue] (externfile2) to[out=south, in=north] (used extern 2.north);
264 \draw[->, thick,blue] (externfile3) to[out=south, in=north] ([yshift=0.3mm]used extern 3.north);
265 \end{tikzpicture}
266 \end{tcolorbox}
267 \endgroup
268
269
270
271 \section*{Introduction}
272 \label{sec:intro}
273
274 \begin{tcolorbox}[
275 title={\introboxtitle{What} is externalization and why you might want it?},
276 ]
277 If you have ever worked on a long document full of \TikZ pictures and maybe
278 Forest trees, you have probably had some compilation-enforced coffee breaks
279 --- even on modern computers, compiling pictures, trees and such takes a lot
280 of time. And you might have wondered, why do I need to compile these
281 pictures over and over again? --- after all, I'm not changing them anymore!
282 Enter \emph{externalization}, a mechanism designed to deal precisely with
283 your issue, by saving the produced pictures into separate PDFs and including
284 those PDFs in subsequent compilations --- in no time at all!
285 \end{tcolorbox}
286
287 \begin{tcolorbox}[
288 title={\introboxtitle{Why} yet another externalization package?},
289 ]
290 \TikZ, the popular and all-powerful graphics language for \TeX, ships
291 including an externalization library (described in \PGFmanual{53}). \TikZ's
292 library does an excellent job, but with one caveat. Assume you're using it
293 for the first time (or after a clean-up) on that long document full of \TikZ
294 pictures and Forest trees. It will take ages to produce all the externalized
295 graphics. Why? Because to get you up to speed, your document has to be
296 compiled many many times --- once for each and every externalized picture.
297 Even with \TikZ's advanced mechanism for skipping the parts of the document
298 irrelevant for the picture at hand, the first externalization can be a
299 daunting task.
300 \end{tcolorbox}
301
302 \begin{tcolorbox}[
303 title={\introboxtitle[How]{How} does Memoize save your time?},
304 ]
305 There is a reason why \TikZ uses an entire compilation cycle to produce a
306 single externalized picture: \TeX\ itself can only produce a single PDF per
307 compilation (at least at the moment). Memoize evades this limitation by
308 dumping the externalized pictures right in the middle of the document
309 (ouch!). More precisely, an externalized picture occurs in the PDF twice,
310 first on a special page of its own and then on a regular page, where you
311 intended it to be. The daring dump obviously necessitates a second step of
312 the procedure, when those special pages are \emph{extracted} into separate
313 PDFs, called \emph{externs}, which are then included into the document in
314 subsequent compilations.
315
316 This two-step procedure, illustrated on the cover page of this manual, is
317 very fast. The first step, which externalizes \emph{all} the pictures into
318 the document itself (the squiggly red arrow), takes virtually no more time
319 than a regular compilation. The time needed for the second step, extraction
320 (the normal red arrows), depends on the system setup, but it ranges from
321 little to almost none.
322 \end{tcolorbox}
323
324 \begin{tcolorbox}[
325 title={\introboxtitle{When} should I use Memoize?},
326 ]
327 In short, whenever you are writing a document containing lots of \TikZ
328 pictures, Forest trees, or other time-consuming constructs, and you are bored
329 waiting for the compilation to finish.
330
331 Using Memoize on a paper containing a single picture does not make much of a
332 difference. But with more complex documents, the speed-up can be immense.
333 For example, the compilation time of my 400-page book containing about 160
334 Forest trees was reduced by more than half, and the compilation time of a
335 260-page Beamer presentation with a hundred complex dynamic trees went from
336 four minutes to a mere half minute!
337 \end{tcolorbox}
338
339
340 \begin{tcolorbox}[
341 title={\introboxtitle{How} much extra work does Memoize require?},
342 ]
343 In principle, none.
344
345 For one, while allowing for manual memoization of selected document chunks
346 (\S\ref{sec:tut:memoize}), Memoize features a system which automatically
347 triggers memoization at each invocation of selected commands and
348 environments. Out of the box, Memoize \emph{automemoizes} \TikZ
349 pictures\noprint{\refenv{tikzpicture}} and Forest
350 trees\noprint{\refenv{forest}}, but the author can easily submit (almost) any
351 command or environment to automemoization (\S\ref{sec:tut:automemoization}).
352 Memoize also does its best to automatically prevent memoization of code that
353 cannot be externalized, like \TikZ pictures with \refkey{/tikz/remember
354 picture}, and to abort memoization in case the memoized code yields any
355 errors.
356 \end{tcolorbox}
357
358
359 \begin{tcolorbox}[
360 title={\introboxtitle{Why} is Memoize not called Externalize?},
361 ]
362 Fundamentally, Memoize is about producing and utilizing \emph{memos} ---
363 pieces of \hologo{TeX} code replicating the effect of the compilation of a
364 document chunk in a computationally less intensive manner. Typically, each
365 memo has an associated extern, which is where the effect of
366 \emph{typesetting} is stored, but conceptually, memos come first. For
367 example, the extern is included back into the document by the memo, and a
368 memo may be associated with any number of externs, including zero.
369
370 Memos solve several externalization-related problems in a generic fashion,
371 allowing for a multitude of applications. For example, they store the
372 information about the associated externs, so that an extern can be integrated
373 back into the document as a box with the original orientation, width, height
374 and depth. They solve the problem of cross-referencing from and into the
375 memoized code by storing its \emph{context} (\S\ref{sec:cross-referencing})
376 and replicating any \cs{label}s which occur in it. They are also crucial for
377 externalizing pictures in Beamer frames overlay by overlay.
378
379 Incidentally, the term ``memoization'' is used with some programming
380 languages to refer to the process of remembering the result of the function,
381 along with the given arguments, so that on subsequent invocations of the
382 function with the same arguments, the result can be returned from memory
383 rather than recomputing it. I would say that, give or take the functions,
384 what Memoize does fits the bill.
385 \end{tcolorbox}
386
387
388 \begin{tcolorbox}[
389 title={\introboxtitle{How} can I make my command (auto)memoizable?},
390 ]
391 Any command which interacts with the rest of the document, like a command
392 which produces a float, must receive special treatment. Some issues can be
393 resolved from within Memoize. Other issues require a memoization-compatible
394 (re)implementation of the command. It is in the hope that package writers
395 will adapt their ``difficult'' commands to Memoize that this package offers a
396 documented interface to the memoization process, fully described in
397 sections~\ref{sec:tut:package}, \ref{sec:memoization-drivers}
398 and~\ref{sec:tut:automemoization-details}.
399
400 An advanced user might also want to know that Memoize ships with two
401 auxiliary packages, which form the base of Memoize's automemoization feature.
402 Package Advice implements a generic framework for extending the functionality
403 of selected commands and environments, while package CollArgs provides a
404 command which can determine the argument scope of any command whose argument
405 structure conforms to \hologo{LaTeX3} argument specification.
406 \end{tcolorbox}
407
408 \begin{tcolorbox}[
409 title={\introboxtitle{What else} is out there?},
410 ]
411 Not long before submitting Memoize to CTAN, I became aware of another new
412 externalization package, \pkg{robust-externalize}, and it seems that the same
413 happened to the author of that package \Smiley, who found the
414 proof-of-concept version of Memoize, which was available
415 at \href{https://github.com/sasozivanovic/memoize}{GitHub} for a while.
416
417 The key idea behind \pkg{robust-externalize} seems to be to extract the code
418 submitted to externalization into separate files, and add the necessary
419 preamble. While a compilation from scratch takes more time than with Memoize
420 (but less than with \TikZ library), the approach allows for parallel
421 compilation of externs --- nice!
422 \end{tcolorbox}
423
424 \newpage
425 \tableofcontents
426
427
428 \section{Before you start}
429 \label{sec:setup}
430
431 \subsection{Installing the extern extraction software}
432 \label{sec:install}
433
434 Good news: using Memoize can be as easy as writing
435 \EmphVerbatim{\usepackage{memoize}} in the preamble.
436
437 Bad news: Memoize won't work out of the box. For optimal operation, it
438 requires additional software. The culprit is extern extraction --- the process
439 which ships the externalized graphics from the main document into separate
440 extern files; for details, see the title page illustration and the
441 \hyperlink{How}{``How''} box in the Introduction. But there's a silver
442 lining: once Memoize is set up, it is set up for good.
443
444 \begin{tcolorbox}[title=What do I have to do?]
445 In principle, all you have to do for Memoize to work \emph{under the default
446 configuration} is install (\href{https://www.perl.org}{Perl} and) Perl
447 library \hreftt{https://metacpan.org/pod/PDF::API2}{PDF::API2} by following
448 the installation guidelines below.
449 \end{tcolorbox}
450
451 Once you have set up your system, I advise you to follow the instructions in
452 section~\ref{sec:tut:test} to test if the setup was successful.
453
454 \paragraph{Installing Perl}
455 If you're running GNU/Linux or macOS, Perl is almost certainly already
456 installed on your system; if it is not, install it using your system's package
457 manager.
458
459 On Windows, you will most likely have to install Perl, if you haven't done so
460 yet. I have tested Memoize with \href{https://strawberryperl.com}{Strawberry
461 Perl}; see \url(https://){www.perl.org} for other options.
462
463 Having to install Perl on Windows also applies to users of \TeXLive;
464 the built-in Perl won't suffice, as Memoize requires a PDF processing library
465 not shipped with \TeXLive. Furthermore, you will also have to convince
466 \TeXLive to use the so-called external Perl; from the command prompt
467 (|cmd.exe|):
468
469 \begin{tcolorbox}[fontupper=\small]
470 |tlmgr conf texmf TEXLIVE_WINDOWS_TRY_EXTERNAL_PERL 1|
471 \end{tcolorbox}%
472
473 \paragraph{Installing the PDF processing library}
474 Once Perl is installed on your system, you will also need to install the PDF
475 processing library \hreftt{https://metacpan.org/pod/PDF::API2}{PDF::API2} (or
476 its fork, \hreftt{https://metacpan.org/pod/PDF::Builder}{PDF::Builder}). On
477 some GNU/Linux distributions, this library is included as a package --- just
478 use your package manager to install it. Otherwise, install it from
479 \hreftt{https://www.cpan.org}{CPAN} using
480 \hreftt{https://metacpan.org/pod/CPAN}{cpan} tool:
481
482 \begin{tcolorbox}[fontupper=\small]
483 |cpan PDF::API2|
484 \end{tcolorbox}%
485
486 \bigskip
487
488 \begin{tcolorbox}[title=The default settings won't cut it for me!]
489 Here are some situations which require alternative settings (with a brief
490 hint):
491 \begin{itemize}
492 \item If you are compiling your documents using \pkg{latexmk}, install Perl
493 and the PDF processing library as detailed above, integrate the example
494 configuration file \reffile{memoize_latexmkrc} (shipped with \pkg{latexmk}
495 since version~4.84) in your |latexmkrc|, and (optionally) set
496 \code{\refmmz{extract}=\refmmz{extract=no}}. Note that
497 \reffile{memoize_latexmkrc} can be adjusted to perform Python-based
498 extraction.
499 \item If you cannot install Perl or one of the required PDF processing
500 libraries, or extern extraction using these libraries fails, install Python
501 and library |pdfrw2| and then use
502 \code{\refmmz{extract}=\refmmz{extract=python}}.
503 \item If neither Perl or Python are an option, use
504 \code{\refmmz{extract}=\refmmz{extract=tex}}. However, this option is slow
505 and requires some fiddling to set up.
506 \end{itemize}
507 For details, consult sections~\ref{sec:extraction} and~\ref{sec:ref:externs}.
508 \end{tcolorbox}
509
510
511 \subsection{The configuration commands}
512 \label{sec:mmzset}
513
514 Memoize can be configured using command \Emph{\refcmd{mmzset}\marg{keylist}}
515 (and friends). The \meta{keylist} argument is a comma-separated list of
516 configuration settings, each setting having the form \meta{key}|=|\meta{value},
517 or sometimes just \meta{key}. The \meta{keylist} argument is processed by package
518 \pkg{pgfkeys} (see \PGFmanual{88}), so you can use all the bells and whistles
519 of that fantastic PGF utility (like easily defining your own styles).
520
521 Here's some examples of \refcmd{mmzset}. For one, to whet your appetite to
522 learn about the various keys in the \refkeypath{/mmz} path, but more
523 importantly now, to show you that white-space is irrelevant in the
524 \meta{keylist} argument, so you can format the keylist as you wish --- as long
525 as it does not contain an empty line (bottom right). I like to add a comma
526 after the final key (bottom left), because if I don't, I often forget to insert
527 it when I add more keys.
528
529 \begin{tcbraster}[raster columns=4, raster valign=bottom]
530 \begin{tcboxedraster}[raster columns=1]{blankest, raster multicolumn=3, valign=bottom}
531 \begin{tcblisting}{listing only}
532 ~\mmzset~{recompile,auto={circuitikz}{memoize},deactivate=\tikz}
533 \end{tcblisting}
534 \begin{tcblisting}{listing only}
535 ~\mmzset~{ auto = \myref {ref}, meaning to context = \foo }
536 \end{tcblisting}
537 \end{tcboxedraster}
538 \begin{tcblisting}{listing only}
539 ~\mmzset~{
540 no memo dir,
541 verbatim,
542 padding=2in}
543 \end{tcblisting}
544 \end{tcbraster}\par
545 \begin{tcbraster}[raster columns=4, raster valign=top]
546 \begin{tcblisting}{listing only, raster multicolumn=2}
547 ~\mmzset~{
548 auto = \qrcode{memoize, args=om},
549 auto = \mycite{cite},
550 padding = 2in,
551 record = makefile,
552 }
553 \end{tcblisting}
554 \begin{tcblisting}{listing only}
555 ~\mmzset~{
556 trace,
557 verb,
558 % comment
559 capture=vbox
560 }
561 \end{tcblisting}
562 \begin{tcblisting}{listing only, bad}
563 ~\mmzset~{
564 recompile,
565 no verbatim,
566
567 prefix=memos/
568 }
569 \end{tcblisting}
570 \end{tcbraster}
571
572 Command \refcmd{mmzset} can be used any time after loading the package. It is
573 very common in the preamble, but also useful in the document body, where its
574 effect is local to the \hologo{TeX} group.\footnote{Any keys with a non-local
575 effect are explicitly marked as such in the reference section.} For example,
576 you can use the idiom on the left below to force recompilation of a single
577 Forest tree somewhere in the middle of the document (in the code listings
578 below, highlighting marks the resulting scope of the \refmmz{recompile}
579 directive). However, as applying some setting to a single piece of
580 automatically memoized code is common, Memoize provides a special command for
581 the occasion: the keys given as the argument of \Emph{\refcmd{mmznext}} will be
582 applied only at the next instance of automemoization, overriding any keys set
583 by \refcmd{mmzset} in case of a conflict. If the command is given more than
584 once, only the final invocation takes effect.
585
586 \begin{tcbraster}[raster columns=2, raster equal height=rows, raster force size=false]
587 \begin{tcblisting}{listing only, mark region={3}{6}}
588 ~{~
589 ~\mmzset~{recompile}
590 \begin{forest}
591 [VP[V][DP]]
592 \end{forest}
593 % ...
594 ~}~
595 \end{tcblisting}
596 \begin{tcblisting}{listing only, add to width=-1em, mark region={3}{5}}
597 % ...
598 ~\mmznext~{recompile}
599 \begin{forest}
600 [VP[V][DP]]
601 \end{forest}
602 % ...
603
604 \end{tcblisting}
605 \end{tcbraster}
606
607 I like to follow |\usepackage{memoize}| by \refcmd{mmzset}, but if you prefer,
608 you can also provide the document-wide configuration as \Emph{package options.}
609 For example, the following are equivalent --- both will force
610 re-externalization of all the externs in the document.\footnote{Note that a
611 slash (\code{/}) may not appear unbraced within package options. While this
612 is usually not an issue, it prevents using \pkg{pgfkeys} key handlers. The
613 workaround is to use key \refmmz{options}.}
614
615 \begin{tcbraster}[raster columns=2, raster equal height=rows]
616 \begin{tcblisting}{listing only}
617 \usepackage{memoize}
618 \mmzset{recompile}
619 \end{tcblisting}
620 \begin{tcblisting}{listing only}
621 \usepackage[recompile]{memoize}
622 \end{tcblisting}
623 \end{tcbraster}
624
625
626 \subsection{The configuration file}
627 \label{sec:memoize.cfg}
628
629 Memoize allows you to configure settings which apply to more than just one
630 document. It does that by attempting to load file \Emph{\reffile{memoize.cfg}}
631 just before executing package options. Given how \hologo{TeX} searches for
632 files, the location of this file determines whether it applies system-wide,
633 user-wide or directory-wide. The directory-wide location is clearly the
634 directory itself. The user-wide and system-wide location depend on the
635 \hologo{TeX} distribution and which format(s) you want to use
636 \reffile{memoize.cfg} with:
637
638 \begin{center}
639 \meta{the relevant texmf tree root}|/tex/|\meta{format}|/memoize/memoize.cfg|
640 \end{center}
641
642 You can say |generic| for \meta{format} if you want the configuration file to
643 be accessible by all formats, otherwise \meta{format} should be one of the
644 formats supported by Memoize: |plain|, |latex| or |context|. The |memoize|
645 subfolder shown above is not obligatory. The texmf root directory depends on
646 the distribution, here's how to figure out what it is:
647
648 \begin{tcboxedraster}
649 [raster columns=1, raster valign=top, raster force size=false]
650 {title=The roots of TEXMF trees}
651 \begin{tcolorbox}[title={\TeXLive[white]\hfill\url[white](https://){tug.org/texlive}}]
652 \begin{description}[itemsep=0pt]
653 \item[user-wide]
654 |tlmgr conf texmf TEXMFHOME|
655 or |kpsewhich -var-value TEXMFHOME|\\
656 the default (on Linux): |/home/|\meta{username}|/texmf|
657 \item[system-wide]
658 |tlmgr conf texmf TEXMFLOCAL|
659 or |kpsewhich -var-value TEXMFLOCAL|\\
660 the default (on Linux): |/usr/local/texlive/texmf-local|
661 \end{description}
662
663 \smallskip
664 Don't forget to run |texhash| or |mktexlsr| after creating \reffile{memoize.cfg}.
665 \end{tcolorbox}
666 \begin{tcolorbox}[title={\MiKTeX[white]\hfill\url[white](https://){miktex.org}},
667 before upper={\parskip0.3\baselineskip plus 2pt\relax},]
668 Open the \MiKTeX Console, select the ``Settings'' page and then the
669 ``Directories'' tab.
670
671 If there is a folder marked with the ``Generic'' purpose with attribute
672 ``User'' (for a user-wide \reffile{memoize.cfg}) or ``Common'' (for a system-wide
673 \reffile{memoize.cfg}), that's the folder you are looking for. Otherwise, create a
674 folder following \MiKTeX's instructions and add it to the list.
675
676 Don't forget to click ``Refresh file name database'' (in the ``Tools'' menu
677 of the \MiKTeX Console) after creating \reffile{memoize.cfg}.
678 \end{tcolorbox}
679 \end{tcboxedraster}
680
681 Note that a directory config file will override the user config, and the user
682 config will in turn override the system-wide one. This should not concern you
683 too much, because you will probably only want to use the user-wide config
684 anyway, which might look something like this:
685
686 \tcbinputexample{memoize-example}[.cfg]{
687 title=\tt memoize.cfg,
688 listing only
689 }
690
691 (The \refmmz{extract} line above ``permanently'' selects the Python-based
692 internal extraction method, which uses script \refscript{memoize-extract.py}.
693 Note that \refmmz{extract} and other extraction-related keys like \refmmz{perl
694 extraction options} only make sense as a package option or as a
695 \refcmd{mmzset} key in \reffile{memoize.cfg}. They will have no effect as a
696 \refcmd{mmzset} key in the document, because extraction happens while the
697 package is loaded.)
698
699 If you want to have a directory-wide configuration based on (rather than
700 overriding) the user-wide configuration, your could write down the ``real''
701 user-wide config in, say, |memoize.user.cfg| (located user-wide), and then
702 |\input| this file by both the user-wide and the directory-wide
703 \reffile{memoize.cfg}, overriding what you want in the latter. Of course, the
704 same logic can be used to base a user-wide config on the system-wide one.
705
706 \section{Your first memoized documents}
707 \label{sec:tut}
708
709 \subsection{Let's see if it works!}
710 \label{sec:tut:test}
711
712 \noprint{\refcmd{tikz}\refenv{tikzpicture}\refenv{forest}}%
713
714 Take example file \texttt{test.tex},\footnote{Where can you find
715 the example files? For one, they are integrated into this manual, so if your
716 PDF viewer supports attachments, you can simply click on the paperclip icon
717 on the top right of the example box (even if you're offline). Otherwise,
718 visit the |examples| subdirectory of wherever you found this document
719 \Smiley. Online, the Memoize documentation can be found at CTAN:
720 \url{https://ctan.org/pkg/memoize}; and if your \hologo{TeX} installation
721 includes the documentation, you should also find it in directory \meta{the
722 root of your \hologo{TeX} installation}|/doc/generic/memoize|.
723
724 Incidentally, while we present a full example document in this section, many
725 code listings will only present the parts of the file relevant for the
726 discussion, for brevity. The example \emph{files}, however, will remain
727 full, compilable documents, including the document preamble etc.} or some
728 simple document containing a \TikZ picture or a Forest tree, add
729 \EmphVerbatim{\usepackage{memoize}} to the preamble,\footnote{Memoize likes to
730 be loaded early. If you get the warning \texttt{Cannot read file
731 \meta{jobname}.pdf}, move \cs{usepackage}\bracestt{memoize} up the
732 preamble; see section~\ref{sec:loading-order} for details.} and compile it
733 twice.
734
735 \begin{itemize}
736 \item The first compilation of the example should produce a three-page PDF.
737 The first two pages are the \emph{extern pages} holding the externalized
738 graphics, while the final page is the (sole page of the) real document. Note
739 that you can see each extern twice: first on a page of its own, and then
740 wherever it belongs to in the real document.
741 \item At the second compilation, the extern pages should have disappeared from
742 the PDF, meaning they were successfully extracted into extern files, which
743 are now embedded into the main document.
744 \end{itemize}
745
746 \tcbinputexample{test}{
747 sidebyside,
748 comment={\centering
749 \includeexamplepdf[extern page,left=2mm,right=2mm]{page=1,trim=1in 1in 1in 1in}\quad
750 \includeexamplepdf[extern page,left=2mm,right=2mm]{page=2,trim=1in 1in 1in 1in}\\[2ex]
751 \includeexamplepdf[document page]{page=3}
752 },
753 }
754
755 You might want to play with this example a bit now.
756 For example, if you reverse the order of the \TikZ picture and the Forest
757 tree, you should notice that the externs don't get recompiled. You won't see
758 any extern pages again until you change the actual code of the picture or the
759 tree --- or until you add some other picture or tree, of course.
760
761 If you \emph{don't} want to automatically memoize \TikZ pictures and/or Forest
762 trees, you can switch off their \emph{automemoization} using key
763 \Emph{\refmmz{deactivate}}. This key takes a list of command and environment
764 names. As you can see below, the command and the environment must be
765 deactivated separately.
766
767 \begin{tcblisting}{listing only}
768 \mmzset{
769 deactivate={\tikz, tikzpicture}, % deactivate automemoization of all TikZ pictures
770 deactivate=\tikz, % deactivate only the command
771 deactivate=tikzpicture, % deactivate only the environment
772 }
773 \end{tcblisting}
774
775
776 \subsection{Memoizing by hand}
777 \label{sec:tut:memoize}
778
779 In the previous section, we have compiled our very first document which used
780 Memoize. In that document, we only had to load the package, as Memoize knows
781 how to externalize \TikZ pictures and Forest trees without any help from the
782 author. But what if you want to externalize some other code? The manual way of
783 doing this is by surrounding the code by a \Emph{\refenv{memoize}} environment,
784 or by making it the argument of the \Emph{\refcmd{mmz}} command. The only
785 difference between the two is that the environment, but not the command,
786 ignores any spaces surrounding the given code.
787
788 \tcbinputexample{manual}{
789 sidebyside, lefthand ratio=0.528,
790 comment={\centering
791 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}\hfill
792 \includeexamplepdf[extern page]{page=2,trim=1in 1in 1in 1in}
793 },
794 }
795
796 Both the \refenv{memoize} environment and the \refcmd{mmz} command take a
797 configuration keylist as the optional argument, so their full syntax is
798 \cs{begin}\braces{\refenv{memoize}}\oarg{keylist}\meta{code to be
799 externalized}\cs{end}\braces{\refenv{memoize}} and
800 \refcmd{mmz}\oarg{keylist}\marg{code to be externalized}. The keys given in
801 this optional argument take precedence over the keys set by \refcmd{mmzset}.
802 Note that \refcmd{mmznext} does not apply to manual memoization.
803
804 Manual memoization is great for one-shot memoizations, but you can use it
805 within your own macros as well. For example, assume that you don't want to
806 externalize \TikZ pictures in general (so you have \refmmz{deactivate}d
807 automemoization of the \cs{tikz} command, as explained at the end of
808 section~\ref{sec:tut:test}), but that you want to easily memoize selected
809 pictures. You could define a memoized variant of the \cs{tikz} command, as shown
810 below (and similarly for the environment). (Note the |%| comment characters in
811 the definition of |\mmztikz|. The definition was intentionally broken into
812 several lines to remind you that the spaces around the argument of \refcmd{mmz}
813 matter.)
814
815 \tcbinputexample{mmztikz}{
816 comment={\centering
817 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}
818 \quad
819 \includeexamplepdf[document page]{page=2}
820 },
821 }
822
823 \clearpage
824 \subsection{Memoizing automatically}
825 \label{sec:tut:automemoization}
826
827 Out of the box, Memoize automatically externalizes \TikZ pictures and Forest
828 trees. Let us see how other commands and environments can be submitted to this
829 \emph{automemoization} process.
830
831 We start with the simpler case of environments (fortunately, externalizing
832 environments also makes sense more often than externalizing commands). You can
833 submit an environment to automemoization by writing
834 \EmphTT{\refmmz{auto}=\marg{environment name}\braces{\refmmzauto{memoize}}} (as
835 a key in \refcmd{mmzset}). The natural (but not the only possible) location
836 for this instruction is the preamble. Below, we automemoize environment
837 \env{circuitikz} of package \pkg{circuitikz}, used for drawing electronic
838 circuits.\footnote{In section~\ref{sec:tut:test}, we learned that
839 automemoization can be switched off using key \refmmz{deactivate}. Memoize
840 also offers key \refmmz{activate}, but you probably won't have to use it, as
841 an \refmmz{auto} declaration automatically activates the submitted command.}
842
843 \tcbinputexample{automemoize-environment}{
844 sidebyside, lefthand ratio=0.6,
845 comment={\centering
846 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in,scale=0.8}
847 },
848 }
849
850 Commands are a bit harder to automemoize, because Memoize cannot possibly know
851 how far the arguments of a command extend (in contrast, the end of an
852 environment is clearly marked). With commands, we must inform Memoize about
853 their argument structure, which we achieve using key \refmmzauto{args} in the
854 second argument of key \refmmz{auto}:
855 \EmphTT{\refmmz{auto}=\meta{command}\braces{\refmmzauto{memoize},
856 \refmmzauto{args}=\marg{argument specification}}}.
857
858 And how does the \meta{argument specification} look like? It is a sequence of
859 letters, each letter determining an argument type. Memoize recognizes the same
860 argument types as \refcmd{NewDocumentCommand}, the \hologo{LaTeX3} command for
861 defining new commands, available in core \hologo{LaTeX} since 2020. For
862 the complete story, see the reference for \refmmzauto{args} in
863 section~\ref{sec:ref:advice} and the frame on page~\pageref{tab:argspec} in
864 section~\ref{sec:ref:collargs}; here, we focus on the two most commonly used
865 types, \docref{xparse:m} and \docref{xparse:o}, and add the optional star for
866 good measure:
867 \begin{center}
868 \begin{tabular}{lll}
869 \toprule
870 letter & argument type & example\\
871 \midrule
872 \docref{xparse:m} & mandatory argument \\
873 & --- either surrounded with braces & |\foo{arg}|\\
874 & --- or a single token & |\foo a|\\
875 \docref{xparse:o} & optional argument, surrounded with brackets & |\foo[arg]|\\
876 \docref{xparse:s} & optional star & |\foo*|\\
877 \bottomrule
878 \end{tabular}
879 \end{center}
880
881 Below, we write |args=om| because command \cs{qrcode} (of package \pkg{qrcode})
882 takes two arguments: a bracketed optional argument, followed by a mandatory
883 argument (in braces).
884
885 \tcbinputexample{automemoize-command}{
886 sidebyside, lefthand ratio=0.603, top=0mm,
887 comment={\centering
888 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}\quad
889 \includeexamplepdf[extern page]{page=2,trim=1in 1in 1in 1in}
890 },
891 }
892
893 If the command we're submitting to automemoization was defined by
894 \refcmd{NewDocumentCommand} or similar, \refmmzauto{args} may be omitted, as in
895 this case, Memoize can retrieve the argument specification on its own.
896
897 This should get you started with automemoization. We'll provide some further
898 basic information in sections~\ref{sec:tut:working-on-a-picture}
899 and~\ref{sec:tut:verbatim} of the tutorial, but only package writers will
900 probably ever need the gory details from
901 section~\ref{sec:tut:automemoization-details}. It is my sincere hope that they
902 will support Memoize in their packages, where necessary, so that you don't ever
903 have to write the \refmmz{auto} declarations. However, there is one thing you
904 should know if you encounter a package supporting Memoize: you should load it
905 \emph{after} Memoize!
906
907
908
909 \subsection{Working on a picture}
910 \label{sec:tut:working-on-a-picture}
911
912
913 Memoize automatically recompiles a picture when the code producing the picture
914 changes. However, sometimes we can modify a picture without changing its code,
915 like when we modify the definition of a command used in the code. In the
916 example below, a predefined style |emph| is applied to the node, producing a
917 node with a red background. Let's say we compile the document (with
918 memoization) and then change this style to set the yellow background.
919 Curiously, the node will remain red.
920
921 \begin{tcbraster}[raster columns=2, raster valign=top, raster column skip=4mm]
922 \tcbinputexample{recompile}(.c1){
923 after title pre={\ (version~1)},
924 comment={\centering
925 \includeexamplepdf[document page]{page=2}
926 },
927 }
928 \tcbinputexample{recompile}(.c2){
929 no attachment,
930 after title pre={\ (version~2)},
931 comment={\centering
932 \includeexamplepdf[document page]{page=1}
933 },
934 }
935 \end{tcbraster}
936
937 The curious thing happens (or rather, doesn't happen) because Memoize doesn't
938 keep track of how commands and styles are defined; it just uses the extern file
939 it created when the old style was in effect. To get a yellow node, we must ask
940 Memoize to reexternalize the picture. The simplest way to do that is by using
941 the \Emph{\refmmz{recompile}} key; below, we write
942 \refcmd{mmznext}\bracestt{\refmmz{recompile}} just before the picture and
943 compile the document again (remember from section~\ref{sec:mmzset} that
944 whatever keys we provide through \refcmd{mmznext} only apply to the next instance of
945 automemoization). After the compilation, we should remove the
946 \refmmz{recompile} directive, otherwise, Memoize will produce the extern page
947 again and again).
948
949 \tcbinputexample{recompile}(.c3){
950 no attachment,
951 after title pre={\ (version~3)},
952 sidebyside, lefthand ratio=0.6,
953 comment={\centering
954 \includeexamplepdf[document page]{page=2}
955 },
956 }
957
958 It is also common to put (again, for the space of a single compilation)
959 \refcmd{mmzset}\braces{\refmmz{recompile}} in the preamble, or to use
960 \refmmz{recompile} as the package option. Either will remake all the
961 externalized graphics in the document, so you can be sure all of them use the
962 latest version of your macros and styles.
963
964 We'll revisit the issue of memoized code depending on macros and styles defined
965 elsewhere in section~\ref{sec:tut:redefinitions}. In this section, we will
966 learn how the issue can be \emph{avoided}, at least to some extent. One idea
967 is to turn off memoization for the picture(s) we are currently working on;
968 another idea is to let Memoize know which definitions the picture relies on.
969 The simplest way to achieve the former is by using key \Emph{\refmmz{disable}};
970 by putting it into a \hologo{TeX} group, we can localize its effect to the
971 selected pictures.
972
973 \tcbinputexample{disable}{
974 sidebyside,
975 lefthand ratio=0.6,
976 top=0mm, % manual top, bottoms and \\[...] to fit the page
977 comment={\centering
978 \includeexamplepdf[extern page,bottom=0.8mm]{page=1,trim=1in 1in 1in 1in}\\[0.5mm]
979 \includeexamplepdf[extern page,bottom=0.8mm]{page=2,trim=1in 1in 1in 1in}\\[0.5mm]
980 \includeexamplepdf[document page,bottom=0.8mm]{page=3}
981 },
982 }
983
984 \begin{tcolorbox}[warning]
985 As you can imagine, key \refmmz{disable} is complemented by \refmmz{enable},
986 but it is perhaps worth mentioning a problem that can arise if you disable
987 memoization for a part of your document by enclosing it in a pair of
988 \refcmd{mmzset}\braces{\refmmz{disable}} and
989 \refcmd{mmzset}\braces{\refmmz{enable}}. Yes, it might work at the moment,
990 but say you later (e.g.\ when you are preparing the final version of the
991 document) decide to disable memoization for the entire document, and say you
992 try to do this by writing \refcmd{mmzset}\braces{\refmmz{disable}} in the
993 preamble. As shown below on the left, you're in for a surprise: memoization
994 will still be enabled in the part of the document following
995 \refcmd{mmzset}\braces{\refmmz{enable}}! The solution is to always disable
996 memoization for a part of the document by using
997 \refcmd{mmzset}\braces{\refmmz{disable}} in a \hologo{TeX} group (i.e.\ the
998 braces), as shown on the right. (In the examples below, the shaded areas
999 mark the parts of the document where memoization is \emph{disabled}.)
1000 \end{tcolorbox}
1001
1002 \begin{tcbraster}[raster columns=2, raster equal height=rows]
1003 \tcbinputexample{disable-bad}{
1004 listing and comment, bad, mark region={4}{7},
1005 comment={\raggedright The upper \refcmd{mmzset}\braces{\refmmz{disable}}
1006 does not have the intended effect,
1007 i.e.\ it doesn't apply to the whole document!}}
1008 \tcbinputexample{disable-good}{
1009 listing and comment, mark region={4}{10},
1010 comment={\raggedright The upper \refcmd{mmzset}\braces{\refmmz{disable}} applies
1011 to the entire document, as expected.}}
1012 \end{tcbraster}
1013
1014 In fact, it might be better to disable memoization using environment
1015 \Emph{\refenv{nomemoize}} or macro \Emph{\refcmd{nommz}}. I also like these
1016 commands because it is easy to add and remove prefix |no| to switch manual
1017 memoization (triggered using environment \refenv{memoize} or macro
1018 \refcmd{mmz}) off and on.
1019
1020 \begin{tcbraster}[raster columns=2, raster equal height=rows]
1021 \tcbinputexample{disable-nomemoize}{listing and comment, mark region={2}{2},
1022 comment=\raggedright Disable using the dedicated environment.}
1023 \tcbinputexample{disable-nommz}{listing and comment, mark region={2}{2},
1024 comment=\raggedright Disable using the dedicated command.}
1025 \end{tcbraster}
1026
1027 It is also possible to disable memoization for all occurrences of a selected
1028 command or environment. In fact, we're already familiar with the procedure
1029 from section \ref{sec:tut:automemoization}, where we used key
1030 \refmmzauto{memoize} inside the second argument of \refmmz{auto} to
1031 automatically memoize all instances of the command or environment given as the
1032 first argument. All we have to do to auto-disable rather than auto-memoize, is
1033 substitute \refmmzauto{nomemoize} for \refmmzauto{memoize}. Note that this
1034 prevents memoization of not only the given command or environment, but also of
1035 any (manual or automatic) memoization which would otherwise occur during its
1036 execution; for example, if |\foo| executes \cs{tikz} under the hood,
1037 autodisabling |\foo| prevents memoization of the inner \cs{tikz}, even though
1038 that command is normally automemoized.
1039
1040 \begin{tcbraster}[raster columns=2, raster equal height=rows]
1041 \tcbinputexample{disable-auto-cmd}{
1042 listing and comment,
1043 mark region={3}{3}, mark region={5}{5},
1044 comment=\raggedright Autodisable within a command.,
1045 }
1046 \tcbinputexample{disable-auto-env}{
1047 listing and comment,
1048 mark region={3}{5},
1049 comment=\raggedright Autodisable within an environment.,
1050 }
1051 \end{tcbraster}
1052
1053 All that said, Memoize actually offers a neater way to switch off the
1054 externalization for the picture I'm currently working on. The
1055 \Emph{\refmmz{readonly}} key instructs Memoize to use whatever externs it had
1056 already produced (thereby reducing the document compilation time), but to
1057 abstain from producing any new externs. In effect, the stuff we are currently
1058 working on does not undergo memoization and therefore does not produce the
1059 clutter which can potentially lead to trouble described in the \refmmz{recompile}
1060 examples above.
1061
1062 What I like to do is load the package using |\usepackage[readonly]{memoize}|,
1063 work on stuff, and once I'm happy with the most recent pictures, remove
1064 \refmmz{readonly} from the package options for one compilation.
1065
1066
1067 \begin{tcbraster}[raster columns=5, raster valign=top, raster column skip=4mm]
1068 \tcbinputexample{readonly}(.c1){
1069 raster multicolumn=2,
1070 after title pre={\ (work in progress)},
1071 comment={\centering
1072 \includeexamplepdf[document page]{page=1}
1073 },
1074 }
1075 \tcbinputexample{readonly}(.c2){
1076 raster multicolumn=3,
1077 no attachment,
1078 title={\tt readonly.tex} (the final version),
1079 comment={\centering
1080 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}\quad
1081 \includeexamplepdf[document page]{page=2}
1082 },
1083 }
1084 \end{tcbraster}
1085
1086 We're now ready to tackle a completely different way of avoiding the issue, by
1087 informing Memoize which definitions the externalized picture depends on. We do
1088 this by appending these definitions to \emph{context} --- when the context of a
1089 picture changes, Memoize recompiles the picture, same as if the code of the
1090 picture itself was changed. (We will talk about context in more detail in
1091 section~\ref{sec:cross-referencing}.)
1092
1093 A command can be added as a dependency using key \Emph{\refmmz{meaning to
1094 context}}. Below, we make the following externalized picture depend on the
1095 definition of macro \cs{answer}; changing this definition will result in the
1096 recompilation of the extern. In general, \refmmz{meaning to context} accepts a
1097 comma-separated list of command and environment names, e.g.\
1098 \code{\refmmz{meaning to context}=\bracestt{\cs{foo},bar}} (note the braces).
1099
1100 Memoize offers several variants of \refmmz{meaning to context}, applicable to
1101 various types of commands. For example, the easiest way of making the picture
1102 depend on the definition of a \pkg{pgfkeys} style is to use key handler
1103 \Emph{\refkey{/handlers/.meaning to context}} --- note the dot in the name, and
1104 observe that \code{emph/.meaning to context} below is executed within
1105 \cs{tikzset} (where key \code{emph} was defined), not \refcmd{mmzset}; see
1106 \PGFmanual{87} to learn about key handlers.
1107
1108 \tcbinputexample{meaning-to-context}(.c1){
1109 after title pre={\ (version~1)},
1110 sidebyside,
1111 comment={\centering
1112 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}
1113 },
1114 }
1115
1116 \tcbinputexample{meaning-to-context}(.c2){
1117 after title pre={\ (version~2)},
1118 sidebyside,
1119 comment={\centering
1120 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}
1121 },
1122 no attachment,
1123 }
1124
1125 All variants of \refmmz{meaning to context} (see reference
1126 section~\ref{sec:ref:memoization:basic} for the full list) may be used within
1127 the externalized picture itself, e.g.\ \code{\cs{node}[emph,
1128 emph/\refkey{/handlers/.meaning to context}]\bracestt{...}} is perfectly
1129 valid --- and also handy when you want to limit the effect of the handler to a
1130 single picture, as \refkey{/handlers/.meaning to context} cannot be used within
1131 \refcmd{mmznext}.
1132
1133
1134
1135 \subsection{Keeping a clean house}
1136 \label{sec:tut:memodir}
1137
1138 Memoize produces lots of auxiliary files. For each piece of memoized code, it
1139 produces two \emph{memo} files (a \emph{c-memo} and a \emph{cc-memo}; we will
1140 learn more about them in section~\ref{sec:memos}), which will be joined by the
1141 extern PDF once extern extraction takes place --- under the default settings,
1142 upon the next compilation. To top it off, changing the memoized code will
1143 produce new memo and extern files, with the old files staying in place.
1144
1145 To reduce the clutter (imagine the directory listing as below, but for a
1146 document with a hundred externalized pictures which you have been working on
1147 for a month) in the output directory (which is normally the directory where you
1148 keep your source files and where the document PDF is created), Memoize keeps
1149 memos and externs in the \emph{memo directory}, named \code{\meta{document
1150 name}.memo.dir} by default. The names of these files consist of one or two
1151 long hexadecimal numbers, followed by extension |.memo| for memos and |.pdf|
1152 for externs. The first hexadecimal number in these filenames is the MD5 sum of
1153 the memoized code and that's how Memoize knows which memo belongs to which
1154 piece of code. The second hexadecimal number is the MD5 sum of the
1155 \emph{context}, a concept crucial for properly externalizing code containing
1156 cross-references and such; we will discuss this concept in detail in
1157 section~\ref{sec:cross-referencing}.
1158
1159 Besides memos and externs, Memoize produces two other auxiliary files, the
1160 \dmmz record file and an extraction log with suffix |.mmz.log|. These files
1161 facilitate the communication between the \hologo{TeX} part of the package and
1162 the extern extraction scripts; see section~\ref{sec:record-files} for
1163 details.\cprotect\footnote{If you're using the \hologo{TeX}-based extraction
1164 (\code{\refmmz{extract}=\refmmz{extract=tex}}), each extern (|.pdf|) is also
1165 accompanied by a \hologo{TeX} log file (|.log|) produced during the
1166 compilation that extracted the extern.}
1167
1168 {%
1169 \relaxmmzcommands
1170 \def\mynewextern clean-house.memo.dir/#1-#2.pdf\relax{%
1171 \forestset{
1172 append={[#1.memo\vphantom{p},pointcolor,name=cmemo]},
1173 append={[#1-#2.memo\vphantom{p},pointcolor,name=ccmemo]},
1174 append={[#1-#2.pdf,pointcolor,name=extern]},
1175 }%
1176 }
1177 \tcbinputexample{clean-house}{
1178 middle=1mm,
1179 comment={%
1180 \begin{tcolorbox}[reset, left=2mm, top=0mm, bottom=0mm, right=0mm,
1181 my boxed title=the folder contents (after two compilations),
1182 attach shifted boxed title to top right,
1183 ]
1184 \begin{forest}
1185 before typesetting nodes={for tree={
1186 font=\scriptsize\tt, grow'=0, folder, s sep=0}},
1187 [\meta{the document folder}, inner xsep=0
1188 [\examplename.tex]
1189 [\examplename.pdf]
1190 [\normalsize\dots, no edge]
1191 [\examplename.mmz, pointcolor, name=mmz]
1192 [\examplename.mmz.log, pointcolor, name=mmzlog]
1193 [\examplename.memo.dir, pointcolor, name=memodir,
1194 TeX={%
1195 \def\mmzNewExtern##1##2##3##4{\mynewextern##1\relax}%
1196 \input{\examplepath.mmz}%
1197 },
1198 ]
1199 ]
1200 \draw[dotted, every node/.style={anchor=west, font=\footnotesize\strut}]
1201 (ccmemo.east) -- +(1.5em,0pt) node(temp){the cc-memo}
1202 (mmz.east) -- (mmz -| temp.west) node{the \dmmz file}
1203 (mmzlog.east) -- (mmzlog -| temp.west) node{the extraction log}
1204 (memodir.east) -- (memodir -| temp.west) node{the memo directory}
1205 (cmemo.east) -- (cmemo -| temp.west) node{the c-memo}
1206 (extern.east) -- (extern -| temp.west) node{the extern}
1207 ;
1208 \end{forest}
1209 \end{tcolorbox}
1210 },
1211 }
1212 }
1213
1214 The clean house (clean in the sense that most of the auxiliary files reside in
1215 a dedicated directory) depicted above is courtesy of key \refmmz{memo dir},
1216 which is in effect by default since Memoize version 1.3.0. Here's how a dirty
1217 house, which we can produce using key \refmmz{no memo dir}, would look like:
1218
1219 {
1220 \relaxmmzcommands
1221 \def\mynewextern dirty-house.#1-#2.pdf\relax{%
1222 \forestset{
1223 append={[\examplename.#1.memo\vphantom{p},pointcolor]},
1224 append={[\examplename.#1-#2.memo\vphantom{p},pointcolor]},
1225 append={[\examplename.#1-#2.pdf,pointcolor]},
1226 }%
1227 }
1228 \tcbinputexample{dirty-house}{
1229 middle=1mm,
1230 comment={%
1231 \begin{tcolorbox}[reset, left=2mm, top=0mm, bottom=0mm,
1232 my boxed title=the folder contents (after two compilations),
1233 attach shifted boxed title to top right,
1234 ]
1235 \begin{forest}
1236 before typesetting nodes={for tree={
1237 font=\scriptsize\tt, grow'=0, folder, s sep=0}},
1238 [\meta{the document folder}, inner xsep=0,
1239 TeX={%
1240 \def\mmzNewExtern##1##2##3##4{\mynewextern##1\relax}%
1241 \input{\examplepath.mmz}%
1242 },
1243 [\examplename.tex]
1244 [\examplename.pdf]
1245 [\normalsize\dots, no edge]
1246 [\examplename.mmz, pointcolor]
1247 [\examplename.mmz.log, pointcolor]
1248 ]
1249 \end{forest}
1250 \end{tcolorbox}
1251 },
1252 }
1253 }
1254
1255 I assume \refmmz{no memo dir} will only be useful in very special situations,
1256 when Memoize is unable to create the memo directory. Presumably, this could
1257 only happen if neither Perl nor Python is available on the system (the author
1258 is then most likely falling back to \hologo{TeX}-based extraction by issuing
1259 \code{\refmmz{extract}=\refmmz{extract=tex}}). For details, see
1260 section~\ref{sec:extraction}.
1261
1262 We have seen that Memoize keeps the document folder reasonably clean by writing
1263 most of its auxiliary files into the memo directory. However, you might
1264 sometimes want to perform some additional clean-up by removing all or certain
1265 memos and externs. Perhaps to return to the blank slate after a major
1266 redefinition of macros which the externalized pictures depend on, perhaps due
1267 to strange errors which you believe might be caused by Memoize (in that case,
1268 please write to me!), or perhaps just to save some disk space --- remember that
1269 typically, many memos and externs are \emph{stale} anyway, i.e.\ they reflect
1270 some previous state of your document and are not needed anymore. Whatever the
1271 reason for the clean-up, rest assured that it is always safe to delete memos
1272 (|.memo|) and externs (|.pdf|s residing next to memos and bearing the same
1273 basename), or even the entire \code{\meta{document name}.memo.dir} directory,
1274 in the sense that you cannot lose data this way.
1275
1276 Stale memos and externs may be deleted without any repercussions whatsoever
1277 (unless you later revert to a previous version of the document, of course). In
1278 fact, you might \emph{want} to delete them periodically, or at least once you
1279 finish writing the document. As it is hard to figure out which memos and
1280 externs are stale, Memoize ships with a clean-up script: writing
1281 \EmphTT{\refscript{memoize-clean.pl}} \meta{document name} (substitute |.py|
1282 for |.pl| to use the Python rather than the Perl version of the script) into
1283 the command line will delete all memos and externs belonging to the document
1284 which are not used in the current version of the document (the script inspects
1285 the \dmmz file to see which memos and externs are currently in use).
1286
1287 To delete all memos and externs, even the ones currently in use, execute the
1288 clean-up script with option \refscript{memoize-clean.pl--all}. Users of
1289 \pkg{latexmk} can this perform this full clean-up by the regular \code{latexmk
1290 -c}, as long as they have integrated the example configuration file
1291 \reffile{memoize_latexmkrc} into their |latexmkrc|.
1292
1293 If you delete a memo or an extern currently in use, you will trigger
1294 recompilation of their code --- so deleting a memo or an extern is actually a
1295 perfectly legal alternative to using the \refmmz{recompile} key! The only
1296 issue with triggering recompilation of a specific extern in this way is that
1297 the extern might be hard to locate. When the number of externs is small
1298 enough, visual inspection via file manager thumbnails works great. When the
1299 number is greater, it might be best to invoke \refmmz{direct ccmemo input} and
1300 use the inverse search function of your \hologo{TeX} editor, but one could also
1301 use \refmmz{trace} or grep the c-memos.
1302
1303 \begin{tcolorbox}[warning]
1304 Deleting memos and externs is never an irreversible operation, as you can
1305 always recreate them, but it is still wise to be cautious when cleaning up.
1306 For one, avoid running \refscript{memoize-clean.pl} after a compilation which
1307 produced errors; a failed compilation can lead to an incomplete \dmmz file,
1308 which can in turn lead to over-deletion. Another bad idea is cleaning up
1309 after disabling Memoize for a part of a document, for the same reason.
1310
1311 All that said, Memoize takes some precautions itself. It will cowardly
1312 refuse to perform the clean-up when the \dmmz file is missing the
1313 end-of-file marker (|\endinput|), assuming that this indicates a fatal error
1314 in the previous compilation. It will do the same in case the \dmmz file is
1315 absent or empty. The latter is assumed to be a result of a globally
1316 \refmmz{disable}d memoization, but note that clean-up will be performed if
1317 memoization was disabled using package \refpkg{nomemoize}: that package does not
1318 touch the \dmmz file, so cleaning up should work as intended.
1319 \end{tcolorbox}
1320
1321
1322
1323 It is also safe to delete the \dmmz file (or any other kind of record file, see
1324 section~\ref{sec:record-files}) residing next to your document's |.pdf|. The
1325 \dmmz file contains the information about which externs should be extracted
1326 from the |.pdf|. Deleting it before this is done (by default, before compiling
1327 the document again) will prevent the extraction (same as if providing the
1328 package option \code{\refmmz{extract}=\refmmz{extract=no}}) and ultimately
1329 result in the recompilation of the externs produced in the previous run.
1330 Deleting it after the extraction will have almost no effect: it will only only
1331 prevent the clean-up script from working (the \dmmz file also lists the
1332 currently active memos and externs, and thereby indirectly informs the clean-up
1333 script which files are stale). For further information on the \dmmz file, see
1334 section~\ref{sec:.mmz}.
1335
1336
1337
1338 \subsection{Writing a book?}
1339 \label{sec:tut:multifile}
1340
1341 Books and other long documents are usually produced from sources which reside
1342 in more than a single file, and to speed up the editing process, authors
1343 usually use some system which allows them to compile each chapter separately.
1344 Can Memoize --- designed for virtually the same task of speeding up the editing
1345 process --- work sensibly in this kind of situation? More precisely, can the
1346 book and the individual chapters share the memos and the externs? Yes they
1347 can! If we instruct Memoize to use the same memo directory for both the book
1348 job and the chapter jobs, then we can externalize graphics when compiling a
1349 chapter and have the externs included when compiling the book (and vice versa).
1350 As shown below,\footnote{Package \pkg{docmute} makes \hologo{LaTeX} ignore the
1351 preamble of the chapter file when including this file into the main
1352 document.} all we need to do is use key \refmmz{memo dir} mentioned in
1353 section~\ref{sec:tut:memodir}. When given a value, \code{\refmmz{memo
1354 dir}=\meta{name}}, it effectively makes Memoize pretend it is working on
1355 document \code{\meta{name}.tex}, in the sense that it stores the memos and
1356 externs in directory \code{\meta{name}.memo.dir}.\footnote{Use key
1357 \refmmz{prefix} if you need need more control over the name and location of
1358 the auxiliary files.} (As you can see below on the left, \meta{name} may
1359 contain a directory part.)
1360
1361 \begin{tcbraster}[raster columns=2, raster valign=top]
1362 \tcbinputexample{book}{listing only}
1363 \tcbinputexample{chapters/chapter1}{listing only}
1364 \end{tcbraster}
1365
1366 In the above example, the individual chapters reside in files stored in the
1367 |chapters| subdirectory, and that's why the |book.tex| preamble uses
1368 \code{\refmmz{memo dir}=chapters/book} (rather than \code{\refmmz{memo
1369 dir}=book} or just \refmmz{memo dir}). However, Memoize has no trouble
1370 with a situation where the main file and the chapters reside in the same
1371 folder; the setup is even simpler, as we may then omit the \refmmz{memo dir}
1372 invocation in the main file. The more complicated situation was chosen to
1373 point out the following potential problem with the setup where the chapters
1374 reside in a subdirectory.
1375
1376 If you're anything like me, you would first go for having a memo directory
1377 immediately contained in the project directory and set up \refmmz{memo dir} as
1378 shown below. Well, this won't work, or at least it won't work with the vanilla
1379 \TeXLive, because \hologo{TeX} will refuse to \emph{write} into (memo)
1380 files outside the directory where it was
1381 executed,\cprotect\footnote{\label{fn:openout_any}In \TeXLive, the
1382 \reffile{texmf.cnf} option controlling this behaviour is called
1383 \docref{openout_any}. By default, it is set to |p| (paranoid), which
1384 ``disallow[s] opening dot files [and] \emph{going to parent directories}, and
1385 restrict[s] absolute paths to be under \docref{TEXMFOUTPUT}'' (emphasis
1386 mine). (In \MiKTeX, the option is called
1387 |[Core]AllowUnsafeOutputFiles|.)} and this is precisely what the chapter
1388 compilation is asked to do below.
1389
1390 \begin{tcboxedraster}[raster columns=2, raster valign=top]{blankest, bad}
1391 \begin{tcblisting}{example title, title=the main file, listing only}
1392 \mmzset{~memo dir=book~}
1393 % ...
1394 \input{chapters/chapter1.tex}
1395 \end{tcblisting}
1396 \begin{tcblisting}{example title, title=a chapter file, listing only}
1397 \mmzset{~memo dir=../book~}
1398 \end{tcblisting}
1399 \end{tcboxedraster}
1400
1401 \pagebreak % manual
1402
1403 Section~\ref{sec:tut:working-on-a-picture} presented some ideas on how to work
1404 on a single picture. Those ideas can be all easily applied to the multi-file
1405 situation. For example, you could use \refmmz{readonly} on the chapter that you're
1406 working on (and that chapter only). This way, the preview of the chapter will
1407 not be tarnished by the extern pages, and if you periodically compile it
1408 without \refmmz{readonly}, or compile the book (which does not have the \refmmz{readonly}
1409 set), you will have a reasonably up-to-date set of externs.
1410
1411 \begin{tcboxedraster}[raster columns=2, raster valign=top]{blankest}
1412 \begin{tcblisting}{example title, title=the main file, listing only}
1413 \mmzset{memo dir=chapters/book}
1414 % ...
1415 \input{chapters/chapter1.tex}
1416 \end{tcblisting}
1417 \begin{tcblisting}{example title, title=the current chapter file, listing only}
1418 \mmzset{memo dir=book, ~readonly~}
1419 \end{tcblisting}
1420 \end{tcboxedraster}
1421
1422 \begin{tcolorbox}[title=For \Emacs users]
1423 I often use this \refmmz{readonly} trick myself, but with a twist. As an Emacs user,
1424 I don't use a \hologo{TeX}-based mechanism (such as the \pkg{docmute} package) to
1425 compile a chapter, but rely on the region compilation feature of Emacs'
1426 AUC\hologo{TeX} package. AUC\hologo{TeX} offers a way to compile the current
1427 buffer (if you don't know what an Emacs buffer is, read ``file'') or region
1428 (roughly speaking, the selected text). It does that by putting the buffer or
1429 the region into a file called |_region_.tex| while dressing it up in the
1430 preamble of the original document (when I'm working on a multi-file document,
1431 it correctly pulls the preamble from the main document). This results in a
1432 compilable region file. My trick is to detect whether I'm compiling a region
1433 (this is the job of |\ifregion|), and if so, put Memoize into the \refmmz{readonly}
1434 mode (an alternative trick would be to \refmmz{disable} it).
1435
1436 This is the trick in a nutshell, but to make it really work we have to
1437 address one further issue: the original document and the region have to share
1438 memos and externs. This happens automatically if the original document
1439 provides an explicit value to \refmmz{memo dir}, e.g.\ if a document called
1440 |doc.tex| contains \code{\refmmz{memo dir}=doc} in the preamble, but I'm lazy
1441 and don't want to write this in every document. Fortunately, the region file
1442 starts with |\message{ !name(|\meta{original document name}|.tex)}| to
1443 indicate the origin. The complicated part of the code below (everything
1444 following \refcmd{mmzset}\braces{\refmmz{readonly}}) parses this header to
1445 extract the \meta{original document name}, which is then fed to \refmmz{memo
1446 dir}. Now, the trick works automatically for any document.\footnote{More
1447 precisely, the trick works for every document where \refmmz{memo dir} is in
1448 effect --- which, remember, is the default. It is easy enough, however, to
1449 adapt the trick to use \refmmz{no memo dir} or an arbitrary
1450 \refmmz{prefix}.}
1451
1452 \tcbinputexample{memoize-region}[.cfg]{
1453 title=\tt memoize.cfg,
1454 listing only,
1455 }
1456 \end{tcolorbox}
1457
1458
1459 \subsection{Writing a presentation?}
1460 \label{sec:tut:beamer}
1461
1462 Memoize ships with built-in support for the most widespread \hologo{LaTeX}
1463 presentation class, Beamer. It allows for externalizing a picture which
1464 changes from overlay to overlay, or which depends on the Beamer mode. However,
1465 loading Memoize in Beamer is a bit peculiar:
1466
1467 \tcbinputexample{beamer}{
1468 comment and listing,
1469 listing options app={lastline=2}, % todo: firstline/lastline --> Makefile
1470 warning, no attachment, title=,
1471 %
1472 comment={Beamer opens the document PDF while loading the class, while
1473 Memoize requires the PDF from the previous compilation intact in order to
1474 extract the externs (when extraction is triggered internally, which is the
1475 default setting). The solution is to load Memoize (a package) before
1476 Beamer (a class), which can be done by using \cs{RequirePackage} instead of
1477 the usual \cs{usepackage}. Easy, if hacky.},
1478 %
1479 }
1480
1481 To memoize a piece of code which produces different results on different
1482 overlays --- by virtue of containing |\pause|, |\only|, and\slash or related
1483 commands --- apply key \Emph{\refmmz{per overlay}}. Without this key,
1484 externalization of the picture will end badly, with a single extern (the final
1485 one) appearing on all overlays. The key may be invoked either from a prior
1486 \refcmd{mmznext} command,\footnote{Of course, \refmmz{per overlay} may also be
1487 invoked from \refcmd{mmzset}, but I guess this won't make sense often. For
1488 example, if you set it for the entire presentation, and the presentation
1489 contains static memoized pictures as well, you will compile those pictures
1490 more times than necessary: once for each overlay, whereas once per frame
1491 would suffice. It might occasionally make sense, however, to use \refmmz{per
1492 overlay} as an \refmmz{auto} option --- consult
1493 section~\ref{sec:tut:verbatim} to learn what that is.} or executed in the
1494 memoized code itself. The example below illustrates the latter option, and
1495 also shows that we may invoke it via its full path,
1496 \refkeypath{/mmz}|/|\refmmz{per overlay}, when listed among \TikZ options, or
1497 any other options processed by \pkg{pgfkeys}.\footnote{Read
1498 section~\ref{sec:per-overlay} to learn how the Beamer support is implemented.
1499 The implementation only uses Memoize's public interface, so understanding it
1500 should help if you need to support some other presentation package.}
1501
1502 \tcbinputexample{beamer}{
1503 listing options app={firstline=3},
1504 middle=1mm,
1505 comment={\def\dpwidth{0.27\linewidth}\def\epwidth{0.15\linewidth}%
1506 \raisebox{-\height}{\includeexamplepdf[extern page,left=0mm,right=0mm]{width=\epwidth,page=1,trim=1in 1in 0.4in 0.6in}}\hfill
1507 \raisebox{-\height}{\includeexamplepdf[document page,left=1mm,right=1mm]{page=2,width=\dpwidth}}\hfill
1508 \raisebox{-\height}{\includeexamplepdf[extern page,left=0mm,right=0mm]{width=\epwidth,page=3,trim=1in 1in 0.4in 0.6in}}\hfill
1509 \raisebox{-\height}{\includeexamplepdf[document page,left=1mm,right=1mm]{page=4,width=\dpwidth}}%
1510 },
1511 }
1512
1513 \begin{tcolorbox}[warning]
1514 If the memoized code changes the value of Beamer's pause counter
1515 |beamerpauses|, e.g.\ by issuing a |\pause|, take care that (i) \refmmz{per
1516 overlay} is executed prior to any changes of |beamerpauses|, and that (ii)
1517 the final value of this counter in the memoized code is the same for all
1518 overlays.
1519 \end{tcolorbox}
1520
1521 Similarly, the externalized picture might depend on the Beamer mode (|beamer|,
1522 |handout|, |trans| or |article|; for details, see \ManualBeamer[part IV of]).
1523 In this case, use \refmmz{beamer mode to prefix}, but note that this key must
1524 be invoked from \refcmd{mmzset} or \refcmd{mmznext}; it won't work if it is
1525 executed in the memoized code.
1526
1527
1528 \subsection{When stuff sticks out}
1529 \label{sec:tut:padding}
1530
1531 Some constructs --- like plain \hologo{TeX}'s \refcmd{llap} and \refcmd{rlap},
1532 and, notably, \TikZ overlays --- fool \hologo{TeX} into thinking that the
1533 ``size'' of the typeset material is different than what it actually is. This
1534 can cause trouble for externalization: a piece of your picture might disappear!
1535 In a sentence, the solution is to manually set the \emph{padding} of the
1536 externs, but let's slow down a bit.
1537
1538 The \TikZ picture in the following example consists of node with a pin on the
1539 right, but let's say we want to horizontally center this picture so that only
1540 the node rather than the entire picture (including the pin) will be centered.
1541 This can be achieved by adding key \refkey{/tikz/overlay} to the pin (actually,
1542 we need to add it to both the pin and its edge). \TikZ normally updates the
1543 extents (called the bounding box) of the picture every time it puts something
1544 in it; when \refkey{/tikz/overlay} is in effect, however, these updates are
1545 temporarily disabled. In effect, the \refkey{/tikz/overlay} key on the pin
1546 below will fool \hologo{TeX} into thinking that the node is all there is to the
1547 picture, so centering will work as desired.
1548
1549 \tcbinputexample{overlay}(.c1){
1550 after title pre={\ (no memoization)},
1551 comment={\centering
1552 \includeexamplepdf[document page]{page=1}
1553 },
1554 }
1555
1556 What happens when we try to externalize this picture? The example below shows
1557 what would happen if Memoize had no concept of \emph{padding} --- which we
1558 simulate by setting \code{\refmmz{padding}=0pt}.\footnote{Unlike in the rest of
1559 the manual, the extern pages in this section are shown without trimming the
1560 whitespace.} Along with the rest of \hologo{TeX}, Memoize would be fooled
1561 into thinking that the picture comprises of the node only, so the pin would
1562 never make it into the extern. You would end up with a document missing the
1563 pin!\footnote{On the first compilation, the document page containing the figure
1564 without padding looks fine, as it uses the result of the compilation rather
1565 than the extern file. But on the second compilation, when Memoize actually
1566 uses the extern, the pin disappears.}
1567
1568 \tcbinputexample{overlay}(.c2){
1569 after title pre={\ (memoization without padding)},
1570 attachment=overlay-no-padding.tex,
1571 sidebyside, lefthand ratio=0.3,
1572 comment={\centering
1573 \includeexamplepdf[size=minimal,boxrule=0.5mm,extern page,
1574 attach boxed title to top right={xshift=1.5mm,yshift=-1mm},
1575 ]{page=1}
1576 },
1577 }
1578
1579
1580 By default, Memoize puts an inch of space around (what it thinks is) the
1581 externalized picture, and if the overlayed parts of the picture fit into this
1582 inch of space, you will find them in the extern and therefore also in the
1583 document. In our example, however, the default padding is not enough --- the
1584 pin is only partially visible.\footnote{You might wonder why I didn't make the
1585 default padding much bigger, like 10 inches. \hologo{TeX} wouldn't be
1586 bothered (unless the resulting extern size exceeded its maximum dimension),
1587 but you might be, because with such a large default padding, all externs
1588 would be huge, most often bigger than the document pages, and remember that
1589 the externs are first dumped into the document, where they can bother you.}
1590
1591 \tcbinputexample{overlay}(.c3){
1592 after title pre={\ (memoization with default padding)},
1593 attachment=overlay-default-padding.tex,
1594 listing and comment,
1595 middle=0.9mm,
1596 comment={\centering
1597 \includeexamplepdf[extern page,size=minimal,boxrule=0.5mm]{page=1}
1598 },
1599 }
1600
1601 The solution is to set the padding manually. Below, I used \refmmz{padding
1602 right} to only increase the padding on the right side (clearly, we also have
1603 \refmmz{padding left}, \refmmz{padding top} and \refmmz{padding bottom}), but
1604 if you're not bothered by a large extern, you can just use
1605 \Emph{\refmmz{padding}}, which sets all four sides at once. By the way, having
1606 too much padding (almost) never hurts, and as you see, you can use (simple)
1607 arithmetic in the value of these keys.
1608
1609 \tcbinputexample{overlay}(.c4){
1610 after title pre={\ (memoization with extra padding)},
1611 attachment=overlay-extra-padding.tex,
1612 listing and comment,
1613 middle=0.9mm,
1614 comment={\centering
1615 \includeexamplepdf[extern page,size=minimal,boxrule=0.5mm]{page=1}
1616 },
1617 }
1618
1619 \enlargethispage{1ex} % manual
1620 Incidentally, the \refmmz{padding} keys only change how
1621 the externalized picture is \emph{stored}. Memoize remembers the size of the
1622 extern as seen by \hologo{TeX} (e.g.\ the bounding box of the picture as
1623 reported by \TikZ, with overlayed parts of the picture protruding out of it),
1624 and it uses that size when integrating the extern into the document --- so
1625 everything works as it should!
1626
1627
1628
1629 \subsection{When things (seem to) go wrong}
1630 \label{sec:tut:verbatim}
1631
1632 Not all code will peacefully submit to memoization, at least not out of the
1633 box. For one, this is the case for code containing a command or environment
1634 which changes the category code of some character(s). Most notably,
1635 \refcmd{catcode}s are changed by so-called verbatim environments like the
1636 standard \hologo{LaTeX} \refenv{verbatim}, but other commands and environments
1637 effect category code changes as well; one example is command \cs{matrix} (of
1638 \TikZ{} library |matrix|). The solution is to use key
1639 \Emph{\refmmz{verbatim}}, which instructs Memoize to collect the code ``as-is''
1640 (in \hologo{TeX}nical terms, under a catcode regime where all characters have
1641 category code 12) and rescan it (with \cs{scantokens}) when starting the
1642 memoization.\footnote{Memoize also offers a \emph{partial} verbatim mode,
1643 triggered by key \refmmz{verb}; in this mode, the braces retain their usual
1644 category codes. Also note that the effect of \refmmz{verbatim} can be
1645 ``undone'' by key \refmmz{no verbatim}.}
1646
1647 Let us illustrate the issue with environment \refenv{tcblisting} (library
1648 |listings| of package \pkg{tcolorbox}), which typesets a code listing alongside
1649 its compiled effect. To successfully manually memoize this environment, we
1650 have to enclose it in a \refenv{memoize} environment containing key
1651 \refmmz{verbatim} in the optional argument --- without
1652 this key, the example below would produce nothing but errors.
1653
1654 \tcbinputexample{verbatim-manual}{
1655 sidebyside,
1656 comment={\centering
1657 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}\\[1ex]
1658 \footnotesize
1659 (The document page is the same as for the \texttt{verbatim-auto} example below.)
1660 },
1661 }
1662
1663 Using \refmmz{verbatim} from \refcmd{mmzset} or \refcmd{mmznext} works just as
1664 well, and the latter can be very useful with automemoization, when some
1665 environment (say, \env{tcolorbox} of package \pkg{tcolorbox}) generally does
1666 not require the verbatim mode, but a specific occurrence does (say, because it
1667 contains a verbatim construct such as \verb!|!\meta{verbatim text}\verb!|! of
1668 the \pkg{ltxdoc} class). However, for an environment such as |tcblisting|, it
1669 makes the most sense to declare it verbatim in general, so that all instances
1670 of the environment will be processed in the verbatim mode. This is simple to
1671 do: add \refmmz{verbatim} to the \refmmz{auto} keylist.
1672
1673 \tcbinputexample{verbatim-auto}{
1674 sidebyside, % lefthand ratio=0.62,
1675 comment={\centering
1676 \footnotesize
1677 (The extern page is the same as for the \texttt{verbatim-manual} example above.)\\[1ex]
1678 \includeexamplepdf[document page]{page=2}\\
1679 },
1680 }
1681
1682 In fact, you can add any \refkeypath{/mmz} key to the \refmmz{auto} keylist,
1683 and the key will be applied to all occurrences of the command or the
1684 environment. For example, adding \refmmz{recompile} to the declaration of
1685 |tcblisting| above would recompile all and only the |tcblisting| environments;
1686 and as an \refmmz{auto} declaration only updates (rather than completely
1687 replaces) a previous declaration, you can also say things like
1688 \code{\refmmz{auto}=\refcmd{tikz}\braces{\refmmz{recompile}}} to recompile all
1689 \TikZ pictures produced by the \cs{tikz} command (handy, as you don't know how
1690 automemoization for \cs{tikz} was declared unless you've read
1691 section~\ref{sec:tut:automemoization-details} or looked at the Memoize's source
1692 code).
1693
1694 One other situation where a piece of code cannot be memoized out of the box is
1695 when it is designed to operate in the vertical mode;\footnote{``Horizontal mode
1696 is typically used to make lines of text; vertical mode is typically used to
1697 stack the lines of a paragraph on top of each other.'' (\TeXbyTopic[\S6])}
1698 examples include the standard \hologo{LaTeX} \refenv{verbatim} environment
1699 (again) and environment \env{algorithmic} of bundle \pkg{algorithms}. The
1700 problem arises because by default, Memoize collects the typeset material into a
1701 horizontal box (\cs{hbox}), i.e.\ in the horizontal mode (note that there is no
1702 way for Memoize to detect the mode required by the memoized code in advance).
1703 Luckily, Memoize provides an easy solution: we simply have to instruct it to
1704 memoize into a vertical box (\cs{vbox}), which we can achieve by writing
1705 \Emph{\refmmz{capture}|=|\refmmz{capture=vbox}}, exemplified below with
1706 environment \env{algorithmic}.\footnote{Note that this solution does not work
1707 for commands and environments which operate in \emph{external} vertical mode;
1708 see section~\ref{sec:limits} for details.}
1709
1710 \tcbinputexample{capture}{
1711 sidebyside, lefthand ratio=0.53, middle=0pt,
1712 comment={\centering
1713 \includeexamplepdf[left=0.9em, right=0.9em]{page=2}
1714 }
1715 }
1716
1717 \newcommand\ProblemsWithMemoization[1][above]{%
1718 Some situations where memoization of otherwise healthy code produces errors
1719 have easy solutions like the two presented #1. In other situations, like
1720 those listed in section~\ref{sec:limits}, memoization is genuinely
1721 impossible. Yet other situations lie somewhere in-between. Memoization is
1722 possible, but implementing it requires some work and knowledge (of the rest
1723 of this manual). To top it off, it is often hard to know which kind of
1724 situation one is dealing with, not least because \hologo{TeX}'s error
1725 messages can be quite uninformative on the subject.
1726
1727 \Href{https://github.com/sasozivanovic/memoize/wiki}{Memoize wiki} on GitHub
1728 lists several commands and environments, marking the issues which they
1729 present to memoization, and providing solutions where possible. The list
1730 will hopefully grow to encompass many commonly used commands, but if the
1731 command you're struggling with is not there, feel free to drop me an email,
1732 or (even better) post a question to
1733 \Href{https://tex.stackexchange.com}{\hologo{TeX} StackExchange} (use tag
1734 \texttt{memoize}).}
1735
1736 \ProblemsWithMemoization
1737
1738 \subsection{The final version of your document}
1739 \label{sec:tut:final}
1740
1741 Bluntly put, you might want to disable Memoize when compiling the final version
1742 of your document, at least if you intend to distribute it in electronic form,
1743 for two reasons:
1744 \begin{itemize}
1745 \item An externalized picture cannot contain hyperlinks. Any hyperlinks (or
1746 hyperlink anchors) contained in the original picture silently disappear
1747 during the production of the extern.
1748 \item When the document contains many externs, the size of the resulting PDF
1749 can be several times the size of the PDF compiled without externalization.
1750 \end{itemize}
1751
1752 Below, we list several ways of fully disabling Memoize. You're of course
1753 already familiar with the first two ways, but what's this \refpkg{nomemoize}
1754 package? The rationale behind this package is that if you want to be
1755 absolutely sure that there is no trace of memoization in your document (for
1756 example, see the \refmmz{disable} -- \refmmz{enable} pitfall in
1757 section~\ref{sec:tut:working-on-a-picture}), the best thing to do is to not
1758 load the package at all. However, you have all those \refcmd{mmzset}s etc.\ in
1759 your source, so the document won't compile without
1760 \cs{usepackage}\braces{\refpkg{memoize}}, right? Right, but wrong. Enter
1761 \Emph{\refpkg{nomemoize}}, a dummy package which accepts all the commands that
1762 Memoize does, but does nothing. In effect, your document will compile, but you
1763 can be sure that not a single memo or extern was loaded or produced.
1764
1765 \begin{tcbraster}[raster columns=3, raster force size=false, raster valign=top]
1766 \begin{tcblisting}{listing only, add to width=2.4em, enhanced}
1767 \usepackage[~disable~]{memoize}
1768 \end{tcblisting}
1769 \begin{tcblisting}{listing only, add to width=-1.6em, enhanced}
1770 \usepackage{memoize}
1771 \mmzset{~disable~}
1772 \end{tcblisting}
1773 \begin{tcblisting}{listing only, add to width=-0.8em, enhanced}
1774 \usepackage{~no~memoize}
1775 \end{tcblisting}
1776 \end{tcbraster}
1777
1778 There is one issue you might need to resolve manually before package
1779 \refpkg{nomemoize} works as intended, though. If you have used any
1780 \refkeypath{/mmz} keys outside \refcmd{mmzset}, you need to list them in
1781 \refcmd{nommzkeys}. For example, if you used \refmmz{per overlay} in the
1782 manner illustrated in section~\ref{sec:tut:beamer}, i.e.\ as
1783 \refkeypath{/mmz}|/|\refmmz{per overlay} among the \TikZ keys, you need to
1784 write \refcmd{nommzkeys}\braces{\refmmz{per overlay}} into the document
1785 preamble. (This command may also appear in a document loading \pkg{memoize},
1786 where it does nothing.)
1787
1788 Another thing you might want to do once you have produced the final version of
1789 the document (in fact, just before you disable Memoize for good) is clean up.
1790 As we saw in section~\ref{sec:tut:memodir}, Memoize produces a lot of auxiliary
1791 files (memos and externs) and it keeps the old versions around! Once your
1792 document is prepared, you can reduce the clutter (and save some disk space) by
1793 deleting memos and externs belonging to the work-in-progress versions of your
1794 document, and keep only those used in the final version. Return to
1795 section~\ref{sec:tut:memodir} to remember how to deploy script
1796 \refscript{memoize-clean.pl} to achieve this.
1797
1798 As the final note, memos and externs (cleaned-up or not) may be copied (along
1799 the document source) to another directory or machine, where they should be
1800 picked up by Memoize. When they are copied to another directory, one should
1801 clearly set \refmmz{memo dir} or \refmmz{prefix} accordingly. There is no need
1802 to copy the \dmmz file (assuming that the document PDF contains no extern pages
1803 waiting for extraction).
1804
1805
1806 \section{Digging deeper}
1807 \label{sec:potential-pitfalls}
1808
1809
1810
1811 \subsection{The limits}
1812 \label{sec:limits}
1813
1814
1815 \paragraph{Line- and page-breaking} An extern can't be broken across lines or
1816 pages.
1817
1818 Externalization of a chunk of code produces a PDF, which is included into the
1819 document at subsequent compilations as a picture --- an unbreakable object (a
1820 horizontal box) with fixed width and height. Therefore, the original code
1821 should produce an unbreakable object as well. For example, this means that you
1822 cannot externalize some text and expect \hologo{TeX} to break it across lines
1823 or pages on subsequent compilations. If you try, the compilation will succeed
1824 --- without an error! --- but your externalized text will end up in a single
1825 line, as shown below.
1826
1827 \tcbinputexample{no-linebreaking}(.c1){
1828 bad,
1829 comment={\centering
1830 \includeexamplepdf[extern page,to be continued on right]
1831 {page=1,trim=1in 1in 14in 1in}\\[1ex]
1832 \includeexamplepdf[document page]{page=2}\\[1ex]
1833 \includeexamplepdf[document page,title=the expected document page](.c2){page=1}
1834 },
1835 }
1836
1837
1838 \paragraph{External vertical mode}
1839
1840 Attempting to memoize a command or environment which operates in
1841 \emph{external} vertical mode (called simply ``vertical mode'' in \TeXbook)
1842 will fail, because Memoize can only operate in \emph{internal} vertical
1843 mode.\footnote{In vertical mode, \hologo{TeX} is ``building the main vertical
1844 list, from which the pages of output are created'', while it is ``building a
1845 vertical list for a vbox'' in internal vertical mode. Similarly, in
1846 horizontal mode, \hologo{TeX} is ``building a horizontal list for a
1847 paragraph,'' while it is ``building a horizontal list for an hbox'' in
1848 restricted horizontal mode. (All quotes are from \TeXbook[85].)}
1849
1850 At one hand, any command or environment which needs interact with the page
1851 breaking mechanism, like a command designed to break its output across pages,
1852 has to operate in external vertical mode. An example of such an environment is
1853 a \env{tcolorbox} with key \refkey{/tcb/breakable} or \refkey{/tcb/float}. On
1854 the other hand, Memoize can only externalize boxes, and to gather typeset
1855 material in a box, it needs to operate either in restricted horizontal mode (by
1856 default) or internal vertical mode (when asked to do so by
1857 \refmmz{capture}|=|\refmmz{capture=vbox}).
1858
1859 In a nutshell, Memoize is inherently incompatible with external vertical mode.
1860 For example, an attempt to memoize a breakable \env{tcolorbox} will fail ---
1861 even if the environment ends up producing a non-broken box. As far as I see,
1862 the only way to support memoization of a command which has to operate in
1863 external mode is from within the package defining the command; the command must
1864 be reimplemented in a Memoize-friendly fashion, perhaps using the ideas
1865 presented in sections~\ref{sec:memoizable-design}
1866 and~\ref{sec:memoization-complex-single-driver}.
1867
1868
1869
1870 \paragraph{\refkey{/tikz/remember picture}}
1871 \TikZ pictures using this key cannot be externalized.
1872
1873 Memoize will silently refuse to externalize any \TikZ picture using
1874 \refkey{/tikz/remember picture} (see \PGFmanual{17.13}). Such pictures
1875 interact with the outside world --- they either reference or are referenced by
1876 other pictures --- and are as such unsuitable for externalization. For
1877 example, while the colored boxes in this manual are generally externalized ---
1878 out of principle \Smiley\ --- the title page illustration is not, and it cannot
1879 be, because of the arrows connecting the various \TikZ pictures composing that
1880 illustration. Some packages use the \refkey{/tikz/remember picture} mechanism
1881 under the hood, and are thus subject to this limitation; one example is package
1882 \pkg{todonotes}, but in general, any package dealing with absolute positions on the
1883 page will be limited in this way.
1884
1885 How does Memoize deal with this situation? Well, by cowardly refusing to
1886 externalize any code which uses \refkey{/tikz/remember picture} or a similar
1887 mechanism for dealing with absolute positions. Luckily, any such mechanism
1888 eventually boils down to the \hologo{TeX} primitive \refcmd{savepos}, so
1889 Memoize hacks --- or as we will say in this manual, \emph{advises} --- this
1890 primitive to abort any ongoing memoization. Initializing and then aborting the
1891 memoization takes some time, to be sure, but the overhead is negligible,
1892 especially in the light of the fact that \emph{not} aborting wreaks real havoc.
1893
1894 Memoize actually provides a user interface for aborting memoization.
1895 Memoization can be aborted either manually, by executing \refcmd{mmzAbort}, or
1896 automatically. The latter is a generalization of the automemoization idea: a
1897 command such as \refcmd{savepos} can be advised to abort memoization by
1898 \refmmz{auto}|=|\refcmd{savepos}\bracestt{\refmmzauto{abort}}. In Memoize,
1899 commands take your advice seriously, so memoization will be aborted whenever
1900 the advised command or environment is encountered.
1901
1902
1903 \paragraph{Indirectly embedded environments} Such environments cannot be
1904 memoized.
1905
1906 Read this if you got an error message such as |Environment "tikzpicture" ended
1907 as "foo"|.
1908
1909 Some environments are defined so that they embed another environment using the
1910 idiom shown on the left: the begin-code of the outer environment opens the
1911 inner environment, and the end-code of the outer environment closes the inner
1912 environment. While this is a fine, and common, idiom, it messes up the
1913 memoization of the inner environment. In the example on the right, trying to
1914 \emph{auto}memoize a \env{minipage} environment (not recommended at all!) causes
1915 trouble with package \pkg{sectionbox}.\footnote{This is a \texttt{Package
1916 collargs Error} because Memoize outsources the actual work of collecting
1917 the environment body to the auxiliary package CollArgs, described in
1918 section~\ref{sec:collargs}.}\fncomma\cprotect\footnote{Why does this happen?
1919 As mentioned in section~\ref{sec:tut:memodir}, Memoize keeps track of memos
1920 and externs by the MD5 sum of the memoized code. But to compute that sum for
1921 an environment, Memoize has to \emph{grab} the environment body, meaning it
1922 has to collect the body in advance. This presents no problem when
1923 \refcmd{end}\meta{environment name} is already present in the input stream at the
1924 time \refcmd{begin}\meta{environment name} is executed, like when you use the
1925 environment normally in your document, or when some macro expands so that it
1926 produces both \cs{begin}\meta{environment name} and \cs{end}\meta{environment
1927 name} simultaneously --- so there would be no problem above if
1928 |\end{minipage}| occurred in the beginning code of \env{sectionbox}. The idiom
1929 presented above is problematic for memoization because at the time \hologo{TeX}
1930 executes |\begin{sectionbox}|, putting |\begin{minipage}| into the input
1931 stream, |\end{sectionbox}| is not yet executed and remains as it is. The
1932 input stream therefore contains a pair of |\begin{minipage}| and
1933 |\end{sectionbox}|. In the normal, non-memoizing course of events this
1934 would not be a problem, because |\end{sectionbox}| would eventually expand to
1935 |\end{minipage}|. During memoization, however, this \emph{is} a problem,
1936 because, as we said, Memoize needs to grab the environment body: upon
1937 encountering |\begin{minipage}|, it looks in the input stream for
1938 |\end{minipage}| --- but there is no |\end{minipage}| in the input stream,
1939 there is only |\end{sectionbox}|, and this results in the |Environment
1940 "minipage" ended as "sectionbox"| error.}
1941
1942 \begingroup
1943 \mmzset{disable}
1944 \begin{tcbraster}[raster equal height]
1945 \begin{tcblisting}{listing only, bad, valign=center}
1946 \newenvironment{foo}% ... args
1947 {% the begin-code of foo
1948 % ...
1949 \begin{tikzpicture}
1950 % ...
1951 }
1952 {% the end-code of foo
1953 % ...
1954 \end{tikzpicture}
1955 % ...
1956 }
1957 \end{tcblisting}
1958 \tcbinputexample{sectionbox}{
1959 bad,
1960 comment={\small\tt
1961 ! Package collargs Error: Environment\\"minipage" ended as "sectionbox".
1962 },
1963 }
1964 \end{tcbraster}
1965 \endgroup
1966
1967 What are your options in this kind of a situation?
1968 \begin{enumerate}
1969 \item The only way to perform any memoization here is to memoize the
1970 \emph{outer} environment --- if that makes sense.\footnote{This avoids the
1971 error because Memoize grabs and memoizes the outer environment, and while
1972 it is memoizing it, further memoization is switched off.} You can do this
1973 either on a case-by-case basis, by enclosing it in the \refenv{memoize}
1974 environment, or automemoize it: \code{\refmmz{auto}=\marg{outer
1975 environment}\braces{\refmmzauto{memoize}}}.
1976 \item But what if memoizing the outer environment is out of the question?
1977 Then, the only way to avoid the error is to prevent the automemoization of
1978 the inner environment.
1979 \begin{enumerate}
1980 \item If you are facing a single occurrence of the problem, it is perhaps
1981 easiest to use key \refmmz{disable} just before the start of the outer
1982 environment.
1983 \item Otherwise, you can automatically disable memoization for the span of
1984 each occurrence of the outer environment: \code{\refmmz{auto}=\marg{outer
1985 environment}%
1986 \braces{\refmmzauto{nomemoize}}}.
1987 \item To deactivate the automemoization of the inner environment for the span
1988 of the outer environment, but otherwise allow for memoization inside the
1989 outer environment: \code{\refmmz{auto}=\marg{outer environment}%
1990 \braces{\refmmzauto{noop}, \refmmz{deactivate}=\meta{inner
1991 environment}}}. Key \refmmzauto{noop} does nothing but apply the
1992 given options (in this case, \refmmz{deactivate}=\meta{inner environment})
1993 to the advised command or environment.
1994 \end{enumerate}
1995 \end{enumerate}
1996
1997 \subsection{Extraction methods and modes}
1998 \label{sec:extraction}
1999
2000 Remember that in Memoize, externalization is a two-step process. First,
2001 externs are typeset on separate pages of the main document, called extern
2002 pages. Then, these extern pages are extracted out of the main document PDF
2003 into extern files, using a program external to \hologo{TeX}. The process is
2004 illustrated on the title page; in this section, we learn how and when the
2005 extraction process can be triggered.
2006
2007 Memoize is flexible in terms of which piece of software is used to perform
2008 extern extraction. It ships with three \emph{extraction methods}:
2009
2010 \begin{description}
2011 \item[\refmmz{extract=perl}] A Perl script, \refscript{memoize-extract.pl}.
2012 This method is the default because it is fast and because Perl is already
2013 installed on many systems running \hologo{TeX}. However, you will most
2014 likely still need to install the PDF processing library
2015 \hreftt{https://metacpan.org/pod/PDF::API2}{PDF::API2}, or its fork
2016 \hreftt{https://metacpan.org/pod/PDF::Builder}{PDF::Builder}. This is as
2017 easy as executing \code{cpan PDF::API2} on the command line.
2018 \item[\refmmz{extract=python}] A Python script, \refscript{memoize-extract.py}.
2019 This method is even faster than the Perl script, though not by much. Try it
2020 out if you have problems installing Perl or the required libraries, or if the
2021 Perl script chokes on your document (see section~\ref{sec:troubleshooting}
2022 for the list of known issues). Besides \href{https://www.python.org}{Python}
2023 ($\geq 3.8$), you will also need the Python library
2024 \hreftt{https://pypi.org/project/pdfrw2/}{pdfrw2}, or its predecessor
2025 \hreftt{https://pypi.org/project/pdfrw/}{pdfrw}. Once again, the
2026 installation is a breeze: |pip install pdfrw2|.
2027 \item[\refmmz{extract=tex}] \hologo{TeX}-based extraction requires no
2028 additional software, but it is much slower than the scripts. As \hologo{TeX}
2029 can only produce a single PDF per compilation, an instance of \hologo{TeX}
2030 (loading the entire document PDF) has to be invoked for each extern, and this
2031 takes time (although the entire process is still much faster than the
2032 venerable \TikZ externalization library).
2033 \end{description}
2034
2035 Memoize is also flexible in terms of how extern extraction is triggered,
2036 providing two \emph{extraction modes}:
2037
2038 \begin{description}
2039 \item[internal]\label{item:setup:who} By default, extern extraction is
2040 triggered internally, i.e.\ by Memoize during the compilation of the
2041 document; more precisely, any externs produced in a compilation are extracted
2042 during the next compilation. To choose an extraction method other than the
2043 default Perl script, load Memoize with the package option
2044 \EmphTT{\refmmz{extract}=\meta{extraction method}}, e.g.\
2045 \code{\refmmz{extract}=\refmmz{extract=python}}.
2046
2047 \item[external] Loading Memoize with package option
2048 \EmphTT{\refmmz{extract}=\refmmz{extract=no}} instructs Memoize to \emph{not}
2049 trigger the (internal) extraction. Now Memoize expects you to trigger the
2050 extraction yourself, in any way that is convenient to you: manually from the
2051 command line, or automatically through your editor, a Makefile, etc. One
2052 particularly convenient option is \pkg{latexmk}, which supports Memoize
2053 through the example configuration file \reffile{memoize_latexmkrc}. All
2054 Memoize cares about is that the extraction takes place before the next
2055 compilation of the document.
2056 \end{description}
2057
2058 Summing up, the extraction mode and method are selected by providing the
2059 appropriate value to key \refmmz{extract}; the possible values are listed in
2060 the Table~\ref{tab:extraction-methods}. Note that unless you're using
2061 \hologo{plainTeX}, this key can only be used as a package option, or in
2062 \refcmd{mmzset} within \reffile{memoize.cfg}. In particular, it is disabled in
2063 the document preamble, because Memoize performs internal extraction while it is
2064 loaded.
2065
2066 \begin{table}[b]
2067 \centering
2068 \begin{tabular}{llll}
2069 \toprule
2070 extraction mode&extraction method&package option&extraction program\\
2071 \midrule
2072 &\refmmz{extract=perl}&\code{\refmmz{extract}=\refmmz{extract=perl}}, or none&\refscript{memoize-extract.pl}\\
2073 internal &\refmmz{extract=python}&\code{\refmmz{extract}=\refmmz{extract=python}}&\refscript{memoize-extract.py}\\
2074 &\refmmz{extract=tex}&\code{\refmmz{extract}=\refmmz{extract=tex}}&|pdftex|\\
2075 \midrule
2076 external &any&(\code{\refmmz{extract}=\refmmz{extract=no}})&any\\
2077 \bottomrule
2078 \end{tabular}%
2079 \caption{Extraction methods}
2080 \label{tab:extraction-methods}
2081 \end{table}
2082
2083 Strictly speaking, external extraction does not require
2084 \code{\refmmz{extract}=\refmmz{extract=no}}. If you extract externs before the
2085 next compilation takes place, and you do this using
2086 \refscript{memoize-extract.pl} or \refscript{memoize-extract.py} (without
2087 option \refscript{memoize-extract.pl--keep}), the extraction script will mark
2088 externs as extracted (by commenting out the relevant lines in the \dmmz file).
2089 The internal extraction, which is triggered at the beginning of the next
2090 compilation, will then find it has nothing to do. Writing
2091 \code{\refmmz{extract}=\refmmz{extract=no}} merely saves you the bit of time
2092 needed for the internal extraction to realize that the externs were already
2093 extracted.
2094
2095 \paragraph{Internal extraction and shell escape mode}
2096
2097 For internal extraction, \hologo{TeX} must be allowed to execute the external
2098 program implementing the chosen extraction method. If you installed Memoize
2099 through the package manager of either \TeXLive or \MiKTeX, your system should
2100 be already set up to allow the execution of Memoize's extraction scripts,
2101 \refscript{memoize-extract.pl} and \refscript{memoize-extract.py}, so
2102 \code{\refmmz{extract}=\refmmz{extract=perl}} and
2103 \code{\refmmz{extract}=\refmmz{extract=python}} should work out of the box.
2104 Otherwise, you can:
2105
2106 \begin{enumerate}[(a)]
2107 \item\label{item:fallback-external} perform the extraction externally
2108 (\code{\refmmz{extract}=\refmmz{extract=no}}),
2109 \item\label{item:fallback-full} compile documents loading Memoize with
2110 command-line option \docref{-shell-escape} (on \TeXLive) or
2111 \docref{--enable-write18} (on \MiKTeX),\footnote{The answers linked
2112 from question ``\href{https://tex.stackexchange.com/q/598818/16819}{How can
2113 I enable shell-escape?}''
2114 on \href{https://tex.stackexchange.com}{\hologo{TeX} StackExchange} will
2115 tell you how you can ask your editor to do this for you.} or
2116 \item\label{item:fallback-restricted} set up the restricted shell escape mode
2117 to allow for the execution of \refscript{memoize-extract.pl} and/or
2118 \refscript{memoize-extract.py}.
2119
2120 On \TeXLive:
2121 \begin{enumerate}[1.]
2122 \item Execute |tlmgr conf texmf shell_escape| to learn which shell escape
2123 mode is currently in effect: |p| = partial = restricted, |f| = disabled and
2124 |t| = enabled.
2125 \item If necessary, use |tlmgr conf texmf shell_escape p| to select the
2126 restricted shell escape mode.
2127 \item Execute |tlmgr conf texmf shell_escape_commands| to get the
2128 \meta{current} list of allowed commands.
2129 \item To add (e.g.) \refscript{memoize-extract.pl} to the list of allowed
2130 commands, execute \code{tlmgr conf texmf shell\_escape\_commands
2131 \meta{current},memoize-extract.pl}.
2132 \end{enumerate}
2133
2134 On \MiKTeX:
2135 \begin{enumerate}[1.]
2136 \item Execute |initexmf --show-config-value=[Core]ShellCommandMode| to learn
2137 which shell escape mode is currently in effect.
2138 \item If necessary, use |initexmf
2139 --set-config-value=[Core]ShellCommandMode=Restricted| to select the
2140 restricted shell escape mode.
2141 \item Execute \code{initexmf
2142 --show-config-value=[Core]AllowedShellCommands[]} to get the
2143 \meta{current} list of allowed commands.
2144 \item To add (e.g.) \refscript{memoize-extract.pl} to the list of allowed
2145 commands, execute \code{initexmf
2146 --set-config-value=[Core]AllowedShellCommands[]=\meta{current};memoize-extract.pl}
2147 \end{enumerate}
2148 \end{enumerate}
2149
2150 The |pdftex| program, employed by extraction method \refmmz{extract=tex}, is
2151 normally not listed among the restricted shell escape mode commands, nor should
2152 it be. If you are forced to use this fallback extraction method, I suggest
2153 going for option~\ref{item:fallback-external} or~\ref{item:fallback-full}
2154 above.
2155
2156 \paragraph{External extraction}
2157
2158 You may use any extraction method\slash program to perform external extraction.
2159 The simplest option is to use the Perl or the Python script,
2160 \refscript{memoize-extract.pl} or \refscript{memoize-extract.py}. To do this
2161 manually from the command line, change into the directory containing your
2162 document, which should contain the auxiliary \dmmz file produced by Memoize,
2163 and execute:
2164
2165 \begin{enumerate}[(a)]
2166 \item \refscript{memoize-extract.pl} \meta{document name}\dmmz \hfill (for
2167 the Perl script)
2168 \item \refscript{memoize-extract.py} \meta{document name}\dmmz \hfill (for
2169 the Python script)
2170 \end{enumerate}
2171
2172 See sections~\ref{sec:.mmz} and \ref{sec:ref:extraction-perl-python} for
2173 further details on the \dmmz file and the extraction scripts.
2174
2175 Things are much more complicated if you want to manually invoke the
2176 \hologo{TeX}-based extraction method, because an instance of |pdftex| needs to
2177 be invoked for each extern, and these have unwieldy names and can be many in
2178 number. In this case, I suggest asking Memoize for help, which it can do by
2179 producing a shell script or a makefile, executing which will extract all the
2180 externs at once. To have Memoize produce a shell script, use key
2181 \code{\refmmz{record}=\refmmz{record=sh}} (or
2182 \code{\refmmz{record}=\refmmz{record=bat}} on Windows); key
2183 \code{\refmmz{record}=\refmmz{record=makefile}} will make a makefile. By
2184 default, these files are named |memoize-extract.|\meta{document name} plus the
2185 |.sh|, |.bat| or |.makefile| suffix.\footnote{If neither a shell script nor a
2186 makefile works for you, you can also define your own kind of \emph{record
2187 file}, to be processed by the external tool of your choice (and
2188 implementation) in order to extract the externs; see
2189 section~\ref{sec:new-record-file} to learn how to do this.}
2190
2191 All that said, triggering the external extraction manually is probably noone's
2192 cup of tea. One appealing option is to use \pkg{latexmk}, which supports
2193 Memoize since version~4.84. All you have to do is integrate the example
2194 configuration file \reffile{memoize_latexmkrc}, shipped with \pkg{latexmk},
2195 into your |latexmkrc|. With this configuration file, |latexmk| will trigger
2196 extraction (using \refscript{memoize-extract.pl} or
2197 \refscript{memoize-extract.py}) and recompile the source when new externs are
2198 detected, and include the auxiliary files created by Memoize in a clean-up when
2199 called with option |-c|. The suggested configuration allows for the aux and
2200 out directory to differ, and may be safely used even with documents which do not
2201 load Memoize.
2202
2203 \paragraph{The memo directory}
2204
2205 Out of the box, Memoize stores memos and externs in a separate directory,
2206 \code{\meta{document name}.memo.dir}, creating this directory if it does not
2207 exist. This behaviour, illustrated in section~\ref{sec:tut:memodir}, is
2208 triggered by key \refmmz{memo dir}, which is in effect by default since Memoize
2209 version~1.3.0.
2210
2211 It is however worth noting that something rather unusual is happening behind
2212 the scenes here. Out of the box, \hologo{TeX} cannot create directories, so
2213 the fact that Memoize \emph{can} create them requires an explanation. You can
2214 create a directory in \hologo{TeX} only by executing an external program which
2215 does that. However, executing the system command |mkdir| is not allowed by
2216 default. This command is considered unsafe, in the sense that it does not
2217 respect the restrictions imposed by a \hologo{TeX} distribution setting called
2218 \docref{openout_any} (in \TeXLive; in \MiKTeX, it is called
2219 |[Core]AllowUnsafeOutputFiles|). Out of the box, \docref{openout_any} is set
2220 to |p| (paranoid), which ``disallow[s] opening dot files [and] going to parent
2221 directories, and restrict[s] absolute paths to be under \docref{TEXMFOUTPUT}.''
2222 Command |mkdir| violates the second clause of this restriction as it can create
2223 a directory anywhere on the system, and this is why |mkdir| is \emph{not} the
2224 initial value of setting \refmmz{mkdir command}, which specifies the system
2225 command used to create the memo directory.
2226
2227 However, Memoize's extraction scripts \emph{are} considered safe in the sense
2228 that they respect restrictions imposed by \docref{openout_any}, so Memoize can
2229 circumvents the |mkdir| issue by assigning these scripts a secondary task:
2230 executing an extraction script with option
2231 \refscript{memoize-extract.pl--mkdir} turns it into a restricted |mkdir|, i.e.\
2232 a ``make directory'' command which respects the restrictions imposed by
2233 \docref{openout_any}. Accordingly, option \refmmz{mkdir command} is
2234 initialized to \code{\refscript{memoize-extract.pl}
2235 \refscript{memoize-extract.pl--mkdir}} (and
2236 \code{\refmmz{extract}=\refmmz{extract=python}} reinitializes it to
2237 \code{\refscript{memoize-extract.pl} \refscript{memoize-extract.pl--mkdir}}).
2238 At the end of the day, it is thus an extraction script which actually creates
2239 the memo directory (in an invocation independent of the internal extraction).
2240 Therefore, Memoize can create the memo directory on any system which allows its
2241 extraction scripts to be executed from \hologo{TeX} itself. Out of the box,
2242 this is the case with both major distributions, \TeXLive and
2243 \MiKTeX.
2244
2245 The default \refmmz{memo dir} regime can therefore fail only on a system where
2246 Memoize's extraction script cannot be executed. Ignoring the case of the user
2247 tightening the security, this can only happen when both Perl and Python,
2248 powering the two variants of the extraction script, are unavailable. This
2249 could presumably happen on Mik\hologo{TeX} on Windows, if the user can install
2250 neither Perl nor Python (note that this does not apply to \TeXLive, as
2251 \code{\refscript{memoize-extract.pl} \refscript{memoize-extract.pl--mkdir}}
2252 works with the built-in Perl). Of course, this situation is undesirable in and
2253 of itself, as it forces the author to fall back to the slower
2254 \hologo{TeX}-based extraction by issuing
2255 \code{\refmmz{extract}=\refmmz{extract=tex}}, but note that it will additionally
2256 prevent the execution of the default \refmmz{mkdir command}. Possible
2257 workarounds include either undoing the effect of \refmmz{memo dir} by issuing
2258 \refmmz{no memo dir}, or changing the \refmmz{mkdir command} to a command like
2259 |mkdir| while allowing \emph{(do this at your own risk)} it to be executed, for
2260 example by compiling the document with \docref{-shell-escape}.
2261
2262
2263
2264 \subsection{From cross-references to the context}
2265 \label{sec:cross-referencing}
2266
2267 Cross-referencing presents a challenge to externalization, because without
2268 special provisions, the ``communication channel'' between the \cs{label} and the
2269 \cs{ref} is broken once we start utilizing the extern.
2270
2271 One direction of the issue occurs when a \refcmd{label} within the memoized
2272 code is referenced by a \cs{ref} on the outside. Without the (built-in)
2273 workaround, the \cs{label} command would only be executed when the extern is
2274 being produced, but not on subsequent compilations of the document, when it is
2275 merely included. Memoize addresses this problem by generalizing
2276 externalization (which can only produce a picture, the extern) to memoization
2277 (which can additionally produce arbitrary code). When Memoize is externalizing
2278 code which contains a \cs{label}, it automatically replicates it into the
2279 \emph{memo}, which is input into the document on subsequent compilations. In
2280 effect, the memo--extern team will continue to produce the label even when it
2281 is utilized rather than compiled. As far as the author is concerned, \cs{label}s
2282 in memoized code ``just work,'' without any observable differences to the
2283 situation without memoization. This is why we will not discuss this direction
2284 of the issue here; a reader interested in how precisely the system works is
2285 invited to read section~\ref{sec:memos}.
2286
2287 The other direction of the issue occurs when a \refcmd{ref} within the memoized
2288 code references \cs{label} on the outside. In this situation, the extern
2289 should be recompiled when the value of the label it refers to changes. Again,
2290 Memoize addresses this problem in full generality, by associating with each
2291 extern a \emph{context}, and recompiling the extern whenever the value of the
2292 context changes.\footnote{The dependency of an extern upon prior definitions
2293 and such can also be addressed in a more \emph{ad hoc} manner, by recompiling
2294 manually; we have already touched upon this subject in
2295 section~\ref{sec:tut:working-on-a-picture}, and will revisit it in
2296 section~\ref{sec:tut:redefinitions}.} All that needs to be done for \cs{ref}
2297 and friends, specifically, is to advise them to add their reference keys to the
2298 context.
2299
2300 As we shall see presently, for the author, the only difference between a
2301 non-memoized and a memoized \cs{ref} is that the latter will take one more
2302 compilation cycle to ``stabilize'' the resulting document. (More precisely,
2303 the memoized situation will take one more cycle if the reference is undefined
2304 on the first compilation.) Then, we will show how we can teach Memoize about
2305 cross-referencing commands other than \cs{ref} and \cs{pageref}. Finally, we will
2306 learn about key \refmmz{context}, the backbone of the cross-referencing support
2307 in Memoize. (The inner workings of the context are further explained in
2308 section~\ref{sec:c-memos}.)
2309
2310
2311 When the memoized code contains a \cs{ref} referring to a label given in another
2312 part of the document, the code is recompiled when (and only when) the reference
2313 changes. Let us look at the following example, jumping in at the point where
2314 it was already compiled enough times that the resulting PDF had stabilized into
2315 a single (document) page with correct references. (Environment
2316 \refenv{nomemoize} disables memoization
2317 of \href{https://ctan.org/pkg/tikzlings}{Ti\emph{k}Zlings}, so that their
2318 externs don't disturb us, and we can focus on the \cs{tikz} command, which does
2319 get externalized and contains a \cs{ref}.)
2320
2321 \tcbinputexample{ref}(.c3){
2322 after title pre={\ (with stable output after three compilations)},
2323 middle=1mm,
2324 comment={\centering
2325 \includeexamplepdf[document page]{page=1}
2326 }
2327 }
2328
2329 Let us add an owl in front of the penguin. In the next compilation, neither
2330 the ``normal'' nor the memoized reference is yet updated, as expected --- in
2331 this compilation, the new value of the penguin label only makes it into the
2332 |.aux| file.
2333
2334 \tcbinputexample{ref}(.c4){
2335 after title pre={\ (after the first compilation with the added owl)},
2336 no attachment,
2337 comment={\centering
2338 \includeexamplepdf[document page]{page=1}
2339 }
2340 }
2341
2342 During the following compilation, the \refcmd{ref}s pick up the new value of the
2343 penguin label, and the \cs{ref} inside the automemoized \cs{tikz} command
2344 forces recompilation of the extern (\emph{how} this is done will be explained
2345 later).
2346
2347 \tcbinputexample{ref}(.c5){
2348 after title pre={\ (after the second compilation with the added owl)},
2349 comment only, no attachment,
2350 comment={\centering
2351 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}\quad
2352 \includeexamplepdf[document page]{page=2}
2353 }
2354 }
2355
2356 In the next compilation, the resulting PDF is finally stabilized, as the
2357 updated extern is (extracted and) included into the document.
2358
2359 \tcbinputexample{ref}(.c6){
2360 after title pre={\ (after the third compilation with the added owl)},
2361 comment only, no attachment,
2362 comment={\centering
2363 \includeexamplepdf[document page]{page=1}
2364 }
2365 }
2366
2367 The message to take home? When some memoized code contains a reference and
2368 that reference changes, it will take three compilation cycles (so, one more
2369 cycle than without memoization) for the resulting document to ``stabilize.''
2370
2371 Out of the box, Memoize supports the standard \hologo{LaTeX} cross-referencing
2372 commands \cs{ref} and \refcmd{pageref}. To automatically recompile code
2373 containing some other cross-referencing command, like \refcmd{vref} of package
2374 \pkg{varioref}, we use the advising framework implemented by package Advice.
2375 This framework is a generalization of automemoization: we use the familiar
2376 \refmmz{auto}, but with advice offered by \Emph{\refmmzauto{ref}} rather than
2377 \refmmzauto{memoize}.
2378
2379 \tcbinputexample{vref}{listing only}
2380
2381 Key \refmmzauto{ref} only works for commands which operate on a single
2382 reference key. However, that single key (which must be enclosed in braces) may
2383 be preceded by optional argument(s) of any kind. Extensions to \refcmd{ref},
2384 e.g.\ the \pkg{hyperref}'s variant, which accepts an optional |*|, work out of
2385 the box. Furthermore, Memoize offers support for cross-referencing commands
2386 which work on multireferences and reference ranges, such as \pkg{cleveref}'s
2387 \refcmd{cref} and \refcmd{crefrange}. Those commands should be advised by
2388 \refmmz{auto} keys \refmmzauto{multiref} and \refmmzauto{refrange},
2389 respectively.
2390
2391 We have jumped into first example of this section with the assumption that it
2392 had already been compiled several times, allowing the resulting PDF to
2393 stabilize. Let us now take a look at what happens at the very first, fresh
2394 compilation of our original example (the one without the owl). (Removing the
2395 |.aux| file before compiling the example again will start afresh.) The curious
2396 thing is that we don't get the extern page containing
2397 \tikz[baseline]\node[draw=red,thick,fill=yellow,anchor=base,font=\bf]{??};.
2398 This is so because by default, Memoize aborts a memoization containing an
2399 undefined reference.
2400
2401 \tcbinputexample{ref}(.c1){
2402 after title pre={\ (after the fresh compilation of the original example)},
2403 comment only, no attachment,
2404 comment={\centering
2405 \includeexamplepdf[document page]{page=2}
2406 }
2407 }
2408
2409 Now sometimes you might want to produce an extern even if it contains an
2410 undefined reference --- for example, you might intend to write the code
2411 containing the \cs{label} much later but enjoy the speed-up offered by Memoize
2412 until then. In that case, apply the \refmmz{auto} key \Emph{\refmmzauto{force
2413 ref}} to \cs{ref}.
2414
2415 \tcbinputexample{ref}(.c7){
2416 after title pre={\ (after the fresh compilation with \refmmzauto{force ref})},
2417 attachment=ref-force.tex,
2418 comment={\centering
2419 \includeexamplepdf[extern page](.c1){page=1,trim=1in 1in 1in 1in}\quad
2420 \includeexamplepdf[document page](.c1){page=2}
2421 }
2422 }
2423
2424 However, note that when you use \refmmzauto{force ref}, \hologo{LaTeX} will
2425 \emph{not} complain about the undefined reference once the extern containing it
2426 is included (unless that reference also occurs in some non-memoized piece of
2427 code). Using \refmmzauto{force ref} is therefore a tiny bit dangerous, and
2428 this is why \refmmzauto{ref}, with the abortion mechanism, is the default
2429 handler for \cs{ref} and \refcmd{pageref}.
2430
2431 As already noted in the previous section, \refcmd{ref} works by appending the
2432 cross-reference to the \emph{context}, the change of which triggers
2433 recompilation. Memoize initializes the context to contain the four
2434 \refmmz{padding}s --- as a result, an extern recompiles if we change the
2435 padding --- but we can append stuff to the context by ourselves, as well.
2436 Below, we use key \Emph{\refmmz{context}} to append the font size (we'll talk
2437 about the value given to this key a bit later); as a result, the picture is
2438 recompiled whenever the font size changes. Below, we change the font size
2439 using command |\small|; changing the default size with a class option such as
2440 |12pt| works as well.
2441
2442 \tcbinputexample{fontsize}(.c1){
2443 sidebyside, lefthand ratio=0.7, after title pre={\ (the first version)},
2444 comment={
2445 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}
2446 }
2447 }
2448
2449 \tcbinputexample{fontsize}(.c2){
2450 sidebyside, lefthand ratio=0.7, after title pre={\ (the second version)},
2451 no attachment,
2452 comment={
2453 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in}
2454 }
2455 }
2456
2457 How does this work? Key \refmmz{context} appends the given tokens to the
2458 \emph{context expression}. When creating an extern or trying to use it,
2459 Memoize (fully) expands this expression and computes the MD5 sum of the
2460 expansion. This \emph{context MD5 sum} then serves as a part of the extern's
2461 filename (see sections~\ref{sec:tut:memodir} and~\ref{sec:memos}). In effect,
2462 Memoize will only find (and utilize) the extern if \emph{the context MD5 sum
2463 computed during (attempted) utilization matches the one computed during
2464 memoization}.
2465
2466 As revealed by looking at the \hologo{LaTeX} source code, \hologo{LaTeX} holds
2467 the current font size in macro |\f@size|, and above, we have effectively added
2468 the contents of this macro to the context. Now, why didn't we simply write
2469 \refmmz{context}|=\f@size|? First, we used |\csname ... \endcsname| because we
2470 were under the normal \hologo{LaTeX} catcode regime, where |@| cannot be a part
2471 of the command name. Of course, we could have temporarily changed the catcode
2472 of |@| using |\makeatletter| and |\makeatother|, but I would advise against
2473 that, because the approach does not work in general: it fails when key
2474 \refmmz{context} is used \emph{within} memoized code (we will explain why in
2475 section~\ref{sec:memos}). Another reason why I recommend the |\csname
2476 ... \endcsname| approach is that it does not result in an error when the
2477 control sequence is not defined (|\csname ... \endcsname| will expand to
2478 |\relax| then); this behaviour is handy for undefined cross-references, for
2479 example. Second, why did I write |fsize={...}| around the control sequence?
2480 Well, because I'm being paranoid, really. Writing \refmmz{context}|={\csname
2481 f@size\endcsname}| would work just as well, but I like to explicitly
2482 ``announce'' the value to prevent any possibility of a conflict with an
2483 alternative context. Imagine that we don't use the ``announcements'' and we
2484 decide to add some other dimension instead of the font size to the context.
2485 Now if that dimension happened to have the same value as the font size, Memoize
2486 would incorrectly pick up the ``font size extern'' instead of producing a new
2487 one.
2488
2489 It bears emphasizing that whatever you add to the context expression must be
2490 fully expandable, and also not merely declared as robust. So writing
2491 \code{\refmmz{context}=\cs{ref}\marg{key}}, for example, would be unwise, since
2492 it would not work as intended when package \pkg{hyperref} is loaded. (This
2493 package declares \refcmd{ref} as robust, so it won't expand to the
2494 cross-reference value.) You have to look up where the cross-references are
2495 stored internally; the cross-reference for \meta{key} turns out to be stored in
2496 the internal control sequence |\r@|\meta{key}, so it is |\csname
2497 r@|\meta{key}|\endcsname| that the \refmmzauto{ref} handler actually appends to
2498 the context.
2499
2500 The padding and font-size contexts are useful quite generally. However, the
2501 context can be pretty command-specific, as well. Consider the \pkg{skak}
2502 package used for typesetting chess games. The board is drawn using command
2503 |\showboard|, but this command has no arguments, because it draws the state of
2504 the board that is reached by the moves given by command |\mainline|. Memoizing
2505 |\showboard| as such will therefore yield the wrong result --- all the boards
2506 will be one and the same board! The solution is to provide the correct
2507 context: we dig into the \pkg{skak} sources and realize that the current
2508 board is stored in macro |\csname chessgame.skak.mainline\endcsname|.
2509
2510 \tcbinputexample{skak}{
2511 sidebyside, lefthand ratio=0.48,
2512 comment={
2513 \includeexamplepdf[extern page]{page=1,trim=1in 1in 1in 1in,scale=0.45}\quad
2514 \includeexamplepdf[extern page]{page=2,trim=1in 1in 1in 1in,scale=0.45}
2515 }
2516 }
2517
2518 If you remove \code{\refmmz{context}=\bracestt{...}} from the code above, you
2519 will end up with a document where the final board drawn takes place of all the
2520 boards. This is so because in that case, all externs are written into the same
2521 file, so the final extern overwrites the previous ones, but note that you will
2522 only observe this after the second compilation, when the externs are actually
2523 used.
2524
2525
2526
2527 \subsection{More on redefinitions and stale externs}
2528 \label{sec:tut:redefinitions}
2529
2530 In this subsection, we elaborate on an issue touched upon at the beginning of
2531 section~\ref{sec:tut:working-on-a-picture}: what happens if the memoized code
2532 depends on some macro or style which gets redefined? The answer was
2533 ``nothing,'' and one solution was to \refmmz{recompile} the code. Let us take
2534 the example from that section a bit further. We will propose no new solution
2535 or workaround, but deepen our understanding of the issue.
2536
2537 \tcbset{
2538 recompile step/.style={
2539 sidebyside, lefthand ratio=0.74, title=,
2540 comment={\includeexamplepdf[blankest][redefinitions](.c\theenumi){#1}},
2541 /utils/exec={\addtocounter{enumi}{1}},
2542 enhanced, overlay app={\node[at=(frame.north west), font=\scriptsize, circle, fill=black!80!white, text=white, inner sep=1pt]{\theenumi};},
2543 },
2544 }
2545 \begin{tcboxedraster}[raster columns=1]{%
2546 title=Working on \texttt{redefinitions.tex}, enhanced, breakable}
2547 \setcounter{enumi}{0}%
2548 \tcbox[blankest]{I like red. My emphasized nodes will have red background.}
2549 \tcbinputexample{redefinitions}(.c1){recompile step={page=2}}
2550 \tcbox[blankest]{Hmm, this particular node is really important,
2551 let me put the text in italics as well!}
2552 \tcbinputexample{redefinitions}(.c2){recompile step={page=2}, no attachment}
2553 \tcbox[blankest]{You know what? Perhaps yellow background would work better
2554 --- in general.}
2555 \tcbinputexample{redefinitions}(.c3){recompile step={page=1}, bad, no attachment}
2556 \tcbox[blankest]{How come my node is still red?! Oh yes, I changed the
2557 style, so I have to recompile the extern!}
2558 \tcbinputexample{redefinitions}(.c4){recompile step={page=2}, no attachment}
2559 \tcbox[blankest]{Ahh, yellow background, that's much better. But you know
2560 what, this double emphasis won't do after all, let me go back to the
2561 upright shape.}
2562 \tcbinputexample{redefinitions}(.c5){recompile step={page=1}, bad, no attachment}
2563 \tcbox[blankest]{Red???!???!? Ok, I know that recompiling will help, but what
2564 happened here?}
2565 \tcbinputexample{redefinitions}(.c6){recompile step={page=2}, no attachment}
2566 \end{tcboxedraster}
2567
2568 What happened is that the externs from steps~1 and~5 share the very same code.
2569 In step~1, this code was compiled when the red |emph| style was in effect, and
2570 that extern lingered and was eventually picked up again in step~5, Memoize
2571 having no idea that it is including an extern produced with the obsolete
2572 definition of the style.
2573
2574 There are two points to this story. First (and forgetting for a moment about
2575 the context, which we started discussing in
2576 section~\ref{sec:cross-referencing}), Memoize identifies externs (and memos) by
2577 the code that produced them --- or more precisely, by the MD5 sum of the code,
2578 as each piece of code has a unique (well, unique enough) MD5 sum. Each extern
2579 is saved in a file whose name contains this MD5 sum; see
2580 section~\ref{sec:tut:memodir} for illustration. Generally, this is a very
2581 useful feature. You can move your picture to another location in the document,
2582 insert some other (externalized) picture in front of it, and so on, all without
2583 triggering recompilation of the extern(s). (None of this is possible with
2584 \TikZ's externalization library, which identifies the externs by the order in
2585 which they appear in the document.)
2586
2587 The downside of the MD5 sum approach is the potential pitfall illustrated
2588 above, and the downside comes about because of the second point of the story:
2589 Memoize does not attempt to delete the ``old'' externs. (However, as described
2590 in section~\ref{sec:tut:memodir}, stale memos and externs can be easily removed
2591 using the \refscript{memoize-clean.pl} script.) That would be not only
2592 dangerous (as any deletion inherently is) but also potentially wasteful: what
2593 if you have only temporarily removed some code, or compiled only a portion of
2594 the document --- you surely wouldn't want your hard-won externs to disappear in
2595 such a situation!
2596
2597 The pitfall described above applies to any command which depends on parameters
2598 which can be set prior to the invocation of the command, like \TikZ pictures,
2599 which depend on the settings given in \cs{tikzset}. After customizing the
2600 settings, you will have to recompile the externs:
2601 \refcmd{mmznext}\braces{\refmmz{recompile}} is useful when you only have to
2602 recompile a single extern; use \refcmd{mmzset}\braces{\refmmz{recompile}}, or
2603 the package option \refmmz{recompile}, to recompile all externs in the
2604 document; and there is also the middle road: if you have changed only Forest's
2605 settings, you can write
2606 \code{\refmmz{auto}=\braces{forest}\braces{\refmmz{recompile}}} to recompile
2607 all and only the Forest trees.
2608
2609 Above, we have seen the ``same code, same extern'' issue manifested ``through
2610 time,'' i.e.\ Memoize was (incorrectly) reusing externs produced in previous
2611 compilations, but the issue can also manifest ``through space.'' This can
2612 happen if the same code appears twice in the same document --- but, crucially,
2613 with some parameters which it depends on changed from one occurrence to the
2614 next. Observe what happens in the following example, where the settings for
2615 |\progressbar| are changed by |\progressbarchange|. After the first
2616 compilation, everything looks fine. But as both extern pages were produced by
2617 the same code, they will be stored into the same file, the second one
2618 overwriting the first one. The second compilation pulls in the externs --- or
2619 rather, the single extern --- resulting in the document containing the second
2620 progressbar in the place of the first one as well.
2621
2622 \tcbinputexample{progressbar}(.c1){
2623 bad, float,
2624 comment={
2625 \begin{tcbitemize}[raster columns=3,raster equal height,halign=center,valign=center]
2626 \tcbitem[title=After 1\textsuperscript{st} compilation]
2627 \includeexamplepdf[extern page,remember as=extern1]{page=1,trim=1in 1in 1in 1in}\\
2628 \includeexamplepdf[extern page,remember as=extern2]{page=2,trim=1in 1in 1in 1in}\\
2629 \includeexamplepdf[document page]{page=3}
2630 \tcbitem[title=After extern extraction\vphantom{p\textsuperscript{nd}}]
2631 \includeexamplepdf[my boxed title=the extern file,attach shifted boxed title to top right,remember as=externfile]{page=2,trim=1in 1in 1in 1in}
2632 \tcbitem[title=After 2\textsuperscript{nd} compilation]
2633 \includeexamplepdf[document page,remember as=doc](.c2){page=1}
2634 \end{tcbitemize}
2635 \begin{tikzpicture}[remember picture,overlay]
2636 \draw[->, thick, red] (extern1.east) to[out=east, in=west] ([yshift=0.5ex]externfile.west);
2637 \draw[->, thick, red] (extern2.east) to[out=east, in=west] ([yshift=-0.5ex]externfile.west);
2638 \draw[->, thick, blue] (externfile.east) to[out=east, in=west] (doc.west);
2639 \end{tikzpicture}
2640 },
2641 }
2642
2643 The same ``extern duplication'' can arise due to how a particular command is
2644 implemented. Say we deactivate automemoization of Forest trees
2645 (\code{\refmmz{deactivate}=forest}), but keep on automemoizing \TikZ pictures.
2646 Forest uses \refenv{tikzpicture}\noprint{\refenv{forest}} under the hood (a
2647 lot); in particular, the tree itself is typeset as a \env{tikzpicture}
2648 environment. But the code that typesets it is the same for all trees,
2649 regardless of their content (the actual content of the tree is hidden in
2650 various macros and boxes, rather than ``pasted'' into the \env{tikzpicture}).
2651 Consequently, the final tree of the document will overwrite all other trees in
2652 the document, just as the second (and thus final) progress bar overwrote the
2653 first one above.\footnote{That is assuming that \hologo{TeX} doesn't simply
2654 spew a bunch of errors. This can happen as well. In the interest of full
2655 disclosure, compiling a Forest tree in the situation described above would
2656 actually also produce --- but only in the first compilation --- a number of
2657 small empty extern pages, one for each node of the tree. A promise: Forest
2658 will soon fully support Memoize and (among other things) avoid this pitfall.
2659 But the principle will remain.} Ouch!
2660
2661 Generally speaking, this final sort of extern duplication issue can arise
2662 whenever we have an ``outer'' command that we don't want to (auto)memoize which
2663 uses an ``inner'' command that we \emph{do} want to automemoize. The solution
2664 is to use the \refmmz{auto} key \refmmzauto{nomemoize} on the outer command;
2665 remember that this key disables memoization for the space of the command or
2666 environment. For example, the correct way to ``deactivate'' automemoization of
2667 \env{forest} environments (but keep automemoizing \TikZ pictures) is
2668 \code{\refmmz{auto}=\bracestt{forest}\braces{\refmmzauto{nomemoize}}}.
2669
2670
2671
2672
2673 \subsection{Supporting Memoize in your package}
2674 \label{sec:tut:package}
2675
2676 \subsubsection{Loading Memoize?}
2677 \label{sec:loading-memoize}
2678
2679 So you want to support Memoize in your package? That's great!
2680
2681 What form precisely this support will take of course depends on your package.
2682 For some commands, a simple \refmmz{auto} declaration will suffice; for other
2683 commands, you might need to write a dedicated \emph{memoization driver}, as
2684 explained in section~\ref{sec:memoization-drivers}. However, one thing is
2685 clear: you \emph{don't} want to require Memoize's presence by
2686 \cs{RequirePackage}\braces{\refpkg{memoize}} in your package. That would
2687 trigger memoization, but triggering memoization should be left at the sole
2688 discretion of the author. The question is, if you're not allowed to load
2689 Memoize, how can you even issue the \refmmz{auto} declaration?
2690
2691 Well, it's not that you really want to \emph{memoize} anything; you want to
2692 make the commands of your package \emph{memoizable}. So:
2693 \cs{RequirePackage}\braces{\Emph{\refpkg{memoizable}}} --- and note that in
2694 `memoizable', the final `e' of `memoize' is dropped, apparently this is the
2695 correct way to spell it.
2696
2697 Loading \refpkg{memoizable} does nothing if Memoize is already loaded, and
2698 behaves like package \refpkg{nomemoize} otherwise --- remember from
2699 section~\ref{sec:tut:final} that \refpkg{nomemoize} is a dummy package which
2700 accepts all the commands that Memoize does, but does nothing.
2701
2702 I have decided to require that Memoize must be loaded before any package that
2703 supports it. Allowing for an arbitrary loading order would complicate the
2704 implementation (and possibly even turn out to be problematic), and furthermore,
2705 Memoize likes to be loaded early anyaway, because it needs to be loaded before
2706 the document PDF is opened if it is to perform the embedded extern extraction.
2707 I don't think the ordering requirement will cause any problems --- let me know
2708 if it does! --- but perhaps it is wise to inform the author about it in the
2709 documentation of your package (I did so at the end of
2710 section~\ref{sec:tut:automemoization}). Anyway, I have enforced the
2711 requirement by raising an error and refusing to load the package in case
2712 Memoize detects \refpkg{memoizable} to be loaded.
2713
2714 Note that the loading order requirement implies that you can use
2715 |\@ifpackageloaded{memoize}| to specifically react to the presence Memoize, if
2716 necessary.
2717
2718
2719 \subsubsection{Memoizable design}
2720 \label{sec:memoizable-design}
2721
2722 Many commands and environments can be submitted to externalization with a
2723 single-line \refmmz{auto} declaration, as illustrated in
2724 section~\ref{sec:tut:automemoization}, perhaps requiring an addition to the
2725 context (section~\ref{sec:cross-referencing}), or some pre- or post-memoization
2726 code (section~\ref{sec:tut:beamer}). In some situations, however, these simple
2727 approaches won't work. Most often, this will happen when the extern must be
2728 integrated into the document in some special way. For example, a command might
2729 internally create floating material, or surround the core typeset material with
2730 some stretchable space.\footnote{Commands and environments of package
2731 \pkg{tcolorbox} exhibit both these issues (see \pkg{tcolorbox} options |float|,
2732 |before| and |after|), and were in fact the inspiration for several technical
2733 details of Memoize.} None of these behaviours can be replicated by merely
2734 including the extern; with respect to the stretchable space, remember that an
2735 extern, being a picture, has fixed size, so if our extra space ended up in the
2736 extern, it would lose the stretchability.
2737
2738 The key to successful memoization of problematic commands is their design. In
2739 a nutshell, the idea is to \emph{break up the command's definition into two
2740 parts, the outer command and the inner command, and only submit the inner
2741 command to automemoization}. We will illustrate this with a simple
2742 environment --- |poormansbox| --- which produces a potentially framed box of
2743 the given width, and surrounds this box with some configurable material --- by
2744 default, this material will be stretchable vertical space, and this will be the
2745 source of our memoization problem. (In terms of user experience, the solution
2746 in this section will leave something to be desired, but we will revisit the
2747 example in section~\ref{sec:memoization-complex-single-driver} and make things
2748 right.)
2749
2750 Let us first take a look at a document using our to-be-developed box
2751 environment. The |poormansbox| environment takes one optional argument, a
2752 keylist of options, which can also be set with the |\pmbset| command. This
2753 being a \textbf{p}oor \textbf{m}an's \textbf{b}ox, it doesn't recognize many
2754 options. One can set the |width| of the box, or request that it occurs in a
2755 |frame|, and the surrounding material can be configured using keys |before| and
2756 |after|. As we will see later in the listing of the package, the box is
2757 |\linewidth| wide by default, has no frame, and is surrounded by vertical glue
2758 |\vskip 2ex plus 1ex minus 1ex| (|2ex| of natural vertical space which may both
2759 stretch and shrink for |1ex|); furthermore, the default value of |before|
2760 contains |\centering| to center the box horizontally (centering is of course
2761 only observable when we change the width of the box).
2762
2763 \tcbinputexample{poormansbox}{
2764 sidebyside, lefthand ratio=0.482,
2765 comment={\centering
2766 \includeexamplepdf[document page,left=0pt,right=0pt]
2767 {page=1,scale=0.5,trim=0mm 3mm 0mm 3mm}
2768 },
2769 }
2770
2771 You might want to play with the example to see that the surrounding vertical
2772 space is indeed stretchable. The example is set up so that the surrounding
2773 space is shrunk a bit to fit all the material onto one page. But if you remove
2774 the final |\lipsum[144]|, the natural amount of all vertical space can be
2775 accommodated on the page, so you should observe an increase of vertical
2776 spacing.
2777
2778 You might have noticed that the example contains nested |poormansbox|es: the
2779 second box (the one which contains |\lipsum[66]|) is nested within the first
2780 one (between |\lipsum[101]| and |\lipsum[75]|). This is intentional: when we
2781 will revisit the |poormansbox| example in
2782 section~\ref{sec:memoization-complex-single-driver}, the implementation will
2783 have to pay special attention to nesting (which presents no problem to the
2784 implementation in this section).
2785
2786 As you can see in the package listing below (\texttt{poormansbox.sty}), the
2787 implementation of our environment is straightforward. We first define the
2788 configuration command |\pmbset| and the option keys (we're using
2789 \pkg{pgfkeys}), and set the option defaults. Then, we move on to the
2790 environment itself: we apply the given options, execute the pre-code, typeset
2791 the box (which is a |minipage| of the given width, potentially wrapped in a
2792 |\fbox|), and execute the post-code.
2793
2794 \tcbinputexample{poormansbox}[.sty]{listing only, float}
2795
2796 Now let's make our |poormansbox| externalizable (\texttt{poormansbox-memoizable.sty}). As
2797 announced above, the idea is to split the definition of the environment into
2798 the outer part (below, the user-level environment |poormansbox|), which
2799 (applies the options and) executes the pre- and the post-code, and the inner
2800 part (below, the macro |\@poormansbox|), which typesets the actual box. If we
2801 then then submit |\@poormansbox|, rather than |poormansbox|, to
2802 automemoization, the outer part will be executed at every compilation (giving
2803 us stretchable space if we request it), while the inner command will be
2804 executed (and memoized) at the first compilation, and substituted for the
2805 extern (the fixed-size box) at subsequent compilations.\cprotect\footnote{You
2806 might have wondered why our definition of the |poormansbox| environment grabs
2807 the body into an argument (|+b|, yielding |#2|), necessitating the use of
2808 |\NewDocumentEnvironment| over the venerable |\newenvironment|. One reason
2809 was that having the environment body as an argument simplifies wrapping the
2810 |\fbox| around the |minipage|, but there is a more important reason. If we
2811 did not grab the environment body, we would have to implement the internal
2812 part of the definition as an environment (|@poormansbox|) as well, and embed
2813 it into the user-level environment using the following idiom:
2814 |\newenvironment{poormansbox}[2][]{...\begin{@poormansbox}}{\end{@poormansbox}...}|.
2815 However, as illustrated in section~\ref{sec:limits}, automemoizing
2816 an environment indirectly embedded in such a way produces an error, because
2817 Memoize is prevented from collecting the environment body.}
2818
2819 \tcbinputexample{poormansbox-memoizable}[.sty]{listing only, float}
2820
2821 Looking at the definition of the internal |\@poormansbox| command, it might
2822 strike you as weird that we have equipped this command with an optional
2823 argument (|#1|) it never uses. However, this optional argument is crucial for
2824 memoization. It will become a part of the memoized code (note
2825 \code{\refmmzauto{args}=om} in the \refmmz{auto} declaration) and thereby
2826 ensure that Memoize will produce separate externs for invocations of
2827 |\@poormansbox| with the same environment body but different options; or in
2828 other words, it will ensure that changing the options recompiles the
2829 extern.\cprotect\footnote{Of course, this only holds for options given in the
2830 optional arguments; if the user changes an option value using a prior
2831 \cs{pmbset} (and that option does not occur in the optional argument),
2832 Memoize won't detect the change. But the end-user knows about this issue, as
2833 it was addressed in sections~\ref{sec:tut:working-on-a-picture}
2834 and~\ref{sec:tut:redefinitions}, and she is also aware of two workarounds:
2835 manual recompilation, or setting the context
2836 (section~\ref{sec:cross-referencing}).
2837
2838 While we're on the subject of the context, note that it is also possible to
2839 deploy context to trigger recompilation of the inner command upon change of
2840 parameters it depends on. We could simply omit the optional argument of
2841 \cs{@poormansbox} and add
2842 \refmmz{context}|={width=\pmb@width,frame=\ifpmb@frame true\else false\fi}|,
2843 to the \refmmz{auto} declaration. The advantage of such an approach is that
2844 Memoize reacts to the change of parameters regardless of whether they are set
2845 using the optional argument or \cs{pmbset}. However, the approach is
2846 unfeasible for commands depending on many parameters: can you imagine listing
2847 all the \TikZ options in the context? Not to mention that a particular
2848 picture usually only depends on a small subset of these options --- by and
2849 large, \TikZ externs would get recompiled too often if the context contained
2850 all \TikZ options.}\footnote{I have toyed with the idea of
2851 splitting (using \pkg{pgfkeys} key filtering) the given options into outer
2852 options, relevant for the outer command, and inner options, relevant for the
2853 inner command, and only passing the inner options to the inner command. The
2854 thought was that would (i) avoid recompiling the extern when only outer
2855 options change, as these options don't affect the inner command, and (ii)
2856 avoid applying the inner options when utilizing the extern, as these options
2857 don't affect the outer command. However, it then hit me that the end-user
2858 might define a style which incorporated both inner and outer options --- I
2859 know I do this with my |tcolorbox|es.}
2860
2861 The downside to automemoizing an internal command is that this might be
2862 counter-intuitive for the author. For example, to deactivate automemoization
2863 of |poormansbox|, the author will have to write
2864 \refcmd{mmzset}\bracestt{\refmmz{deactivate}=\cs{@poormansbox}} (note the
2865 |\@|), but they will have no clue they have to do this unless they have
2866 carefully read |poormansbox|'s documentation. Even worse, the above
2867 \refcmd{mmzset} command will not work unless surrounded by |\makeatletter| and
2868 |\makeatother|, as it refers to an internal control sequence containing |@|.
2869 Well, Memoize offers \refmmz{auto csname}, \refmmz{activate csname} and
2870 \refmmz{deactivate csname}, so that |@| category code manipulations can be
2871 omitted by writing \refcmd{mmzset}\bracestt{\refmmz{deactivate
2872 csname}|=@poormansbox|}, but still.
2873
2874 Another downside could occur when you use the same (automemoized) internal
2875 command in service of several user interface commands. For the sake of
2876 illustration, assume we have also defined an UI-macro |\pmb| which again relies
2877 on |\@poormansbox|. How is the author to deactivate automemoization of |\pmb|
2878 but leave the |poormansbox| environment intact? This is how:
2879 \refcmd{mmzset}\bracestt{\refmmz{auto}=\cs{pmb}\bracestt{\refmmzauto{args}=m,
2880 \refmmzauto{nomemoize}}}. Again, counter-intuitive; the author expects
2881 \refcmd{mmzset}\bracestt{\refmmz{deactivate}=\cs{pmb}} to work.
2882
2883 One other consequence of this approach is that the code included in the c-memo
2884 (if \refmmz{include source in cmemo} is in effect) will not faithfully reflect
2885 the source: as shown in the c-memo listing below, it will contain
2886 |\@poormansbox{...}| instead of |\begin{poormansbox}...\end{poormansbox}| ---
2887 even if this might actually count as an advantage, as the discrepancy will at
2888 least inform the author who refuses to read the fine material accompanying our
2889 |poormansbox| that something funky is going on.
2890
2891 \tcbinputexample{poormansbox-memoizable}[.cmemo]{
2892 listing only,
2893 title=the c-memo of the last \texttt{poormansbox} environment,
2894 no attachment,
2895 }
2896
2897
2898 In a nutshell, automemoizing an internal command might be counter-intuitive for
2899 the author. But the core idea --- to support memoization of a resistant
2900 command by splitting its definition into the outer and the inner command --- is
2901 sound, and we will elaborate on this idea in
2902 section~\ref{sec:memoization-complex-single-driver}, where we will revisit our
2903 |poormansbox| example and develop a variant of this environment which is both
2904 memoizable and user-friendly.
2905
2906
2907 \section{Under the hood}
2908 \label{sec:under-the-hood}
2909
2910 This chapter is written for three audiences: a curious user who wants to
2911 know how Memoize does what it does; a package writer who wants to support
2912 Memoize in a tricky situation; and myself, lest I forget why I made the design
2913 decisions that I made.
2914
2915 \subsection{The entry point}
2916 \label{sec:Memoize}
2917
2918
2919 From the author's perspective, the functionality of this package is entered
2920 either through the manual memoization commands (macro \refcmd{mmz} and
2921 environment \refenv{memoize}), or via automemoization. And while that is
2922 correct, those user interface entry points merely determine what code is
2923 submitted to memoization, and set any options specific to the upcoming
2924 memoization. The real fun starts with command \Emph{\refcmd{Memoize}}, which
2925 is eventually executed by both manual and automatic memoization.
2926
2927 Not every call to \refcmd{Memoize} results in memoization. Calling this macro
2928 has three possible outcomes. It can result in \emph{memoization}, which
2929 produces the memos and externs; in \emph{utilization} of the result of an
2930 earlier memoization (which boils down to inputting the memos); or in
2931 \emph{regular compilation}, whereby the code is compiled
2932 (almost)\footnote{\label{fn:regular-compilation-almost}This is absolutely true
2933 for memoized code which is ``contained'' in the sense of not peeking into the
2934 input stream following the memoized code. In general, code which fails to
2935 satisfy this containment requirement is most likely simply not memoizable;
2936 but there are borderline cases. For example, \refcmd{ignorespaces} at the
2937 end of some code will have the expected effect in the absence of Memoize, but
2938 no effect when executed either during memoization or regular compilation
2939 under Memoize, simply because it will hit some code belonging to Memoize
2940 rather than the continuation of the document. Memoize offers the
2941 \refmmz{ignore spaces} provision to work around this specific problem.} as if
2942 Memoize was not there. Which outcome obtains depends on several factors. The
2943 decision logic is depicted below, and note that you can \refmmz{trace} the
2944 action on the terminal.
2945
2946 \begin{center}
2947 \begin{forest}
2948 yes/.style={edge=->,edge label={node[font=\scriptsize,pos=0.4,left,#1]{yes}}},
2949 no/.style={edge=->,edge label={node[font=\scriptsize,pos=0.4,right,#1]{no}}},
2950 use/.style={content=utilization\\(of the cc-memo)\vphantom{dj},fill=orange,align=center},
2951 memoize/.style={content=memoization\vphantom{dj},fill=green},
2952 compile/.style={content=regular compilation\vphantom{dj},fill=emphcolor},
2953 for tree={
2954 l sep*=1.5,
2955 if n children={0}{}{align=center},
2956 child anchor=north,
2957 },
2958 [Is Memoize enabled?\\(\refcmd{ifmemoize}),
2959 [Are we currently undergoing memoization?\\(\refcmd{ifmemoizing}), yes
2960 [,compile, yes]
2961 [Is the\\\refmmz{recompile}\\mode in effect?, no
2962 [,memoize, yes]
2963 [Do the relevant\\memos and externs exist?, no
2964 [,use, yes]
2965 [Is the\\\refmmz{readonly}\\mode in effect?,no
2966 [,compile, yes]
2967 [,memoize, no]
2968 ]
2969 ]
2970 ]
2971 ]
2972 [,compile, no]
2973 ]
2974 \end{forest}
2975 \end{center}
2976
2977 As the memoization options were already set by the user interface entry points,
2978 you might expect, quite reasonably, that \refcmd{Memoize} takes a single
2979 argument, the code submitted to memoization. After all, what more does it
2980 need? Clearly, executing this code is what produces the typeset material, and
2981 to detect whether the code has ``changed'' (in order to recompile the memos and
2982 externs), we compute the MD5 sum of this very code, don't we? Well, the
2983 reality is a bit more complicated. When it comes to automemoized commands, the
2984 code which the MD5 sum is computed off of (and which is displayed in the c-memo
2985 if \refmmz{include source in cmemo} is in effect) is not exactly the same as
2986 the code we compile (during either memoization or regular compilation). We'll
2987 see what the difference is in section~\ref{sec:tut:automemoization-details};
2988 what matters here is that we must provide \refcmd{Memoize} with both and that
2989 this macro therefore takes two arguments: the \emph{identification code}, which
2990 the MD5 sum is computed off of, and the \emph{executable code}, which, well, is
2991 the code that gets executed during memoization (or regular compilation).
2992
2993 Let's illustrate this with an example which is probably entirely useless (but
2994 don't worry, we'll get to a realistic example in
2995 section~\ref{sec:tut:automemoization-details}). We first memoize some text
2996 manually, using command \refcmd{mmz}, and then do something very stupid: we use
2997 this very text as the identification code for the following \refcmd{Memoize},
2998 even if the executable code of that command is completely different. The
2999 second line of the typeset output should convince you that the first argument
3000 to that command was really used to produce the extern; and one further
3001 compilation should convince you that the first argument was indeed used to
3002 identify the extern: the extern produced by \refcmd{mmz} was overwritten by the
3003 extern produced by \refcmd{Memoize}, in the fashion of the |progressbar|
3004 example from section~\ref{sec:tut:redefinitions}.
3005
3006 \tcbinputexample{memoize-internal}(.c1){
3007 comment={%
3008 \includeexamplepdf[document page,
3009 after title pre={\ (after the first compilation)}
3010 ]{page=3}\hfill
3011 \includeexamplepdf[document page,
3012 after title pre={\ (after the second compilation)}
3013 ](.c2){page=1}
3014 },
3015 }
3016
3017 The example above also illustrates a(nother) peculiar feature of
3018 \refcmd{Memoize}. \refcmd{Memoize} does not open a new \hologo{TeX} group, but
3019 it \emph{expects a group to be opened prior to calling it}, as it will issue an
3020 |\endgroup| at some point. Specifically, the memoization group will be closed
3021 before regular compilation or utilization, but after memoization. If you want
3022 to know why, read the boxed text below.
3023
3024 \begin{tcolorbox}[title=\string\Memoize\ and grouping, enhanced, breakable]
3025 One important desideratum behind the design of Memoize was that using the
3026 package should disrupt the original, Memoize-less compilation as little as
3027 possible. In particular, if the memoized code contains local assignments
3028 whose effect (in the original compilation) persists into the rest of the
3029 document (until the end of the surrounding \hologo{TeX} group, of course),
3030 wouldn't one want these local effects to persist when Memoize is around, as
3031 well? Fortunately, most memoized code does not have persistent local effects
3032 (at least for me, it is usually environments, like \env{tikzpicture}s, that I
3033 want to memoize, and environments introduce a group anyway) --- fortunately,
3034 because there are design reasons for enclosing memoization in a \hologo{TeX}
3035 group (or two), and this enclosure will of course cancel the effect of local
3036 assignments in the memoized code.
3037
3038 For one, the user interface memoization commands, such as \refcmd{mmz} and
3039 automemoized commands, allow for options specific to a particular piece of
3040 memoized code (the options given as the optional argument to manual
3041 memoization, the next-options and the auto-options), and to delimit their
3042 effect, it makes most sense to apply them in a group. I have toyed with the
3043 idea of working around the introduction of a group by manually saving and
3044 restoring all the options, but I quickly gave up on this line of thought.
3045 For one, manually saving and restoring the options would be cumbersome and
3046 error-prone, and probably also slower than using the group. But even worse,
3047 all that work would not really solve the problem of the persistence of local
3048 effects, because memoization itself introduces a group, as well: during
3049 memoization, the typeset material is collected into a box, and opening a box
3050 introduces a group. In some particular situations, this could be avoided by
3051 typesetting the memoized code as-is and collecting the resulting material
3052 using |\lastbox|, but this approach cannot work in general. In general,
3053 memoization will take place in a group, so the issue of local effects must be
3054 addressed in some other way. Memoize offers the following workaround: during
3055 memoization, the memoized code can (globally) add code to the \refmmz{after
3056 memoization} hook, which gets executed immediately after closing the
3057 memoization group.
3058
3059 Does this mean it would be best if the user interface memoization commands
3060 straightforwardly surrounded \refcmd{Memoize} by |\begingroup| and
3061 |\endgroup|? For example, \refcmd{mmz} would open the memoization group, let
3062 \refcmd{Memoize} do its work, and then close the group. Not really.
3063 Remember that memoization is not the only possible outcome of calling
3064 \refcmd{Memoize}. Perhaps we can at least retain the local effects of a
3065 regular compilation, and of utilization?
3066
3067 We can, by finely tuning the timing of the memoization group closure within
3068 \refcmd{Memoize}. This command is designed to close the memoization group
3069 after memoization, but before regular compilation and utilization. Closing
3070 the group after memoization makes sure that the given options are in effect
3071 during this process. By closing the group prior to regular compilation,
3072 regular compilation of the memoized code (which takes place when Memoize is
3073 disabled, for example) is guaranteed to have (almost, see
3074 footnote~\ref{fn:regular-compilation-almost}) exactly the same effect as the
3075 compilation of that code in absence of Memoize; in particular, the effect of
3076 any local assignments will persist into the rest of the document. Finally,
3077 closing the group before utilization simplifies the construction of the memo
3078 in the cases where we need to replicate local effects of the memoized code
3079 --- the group closed, there is no need to smuggle local assignments out of a
3080 memo.
3081 \end{tcolorbox}
3082
3083 \subsection{Memos}
3084 \label{sec:memos}
3085
3086 Up until now, we have pretended that there is a single kind of a memo file. In
3087 truth, there's two kinds: \emph{code memos}, or \emph{c-memos} for short; and
3088 \emph{code--context memos}, or \emph{cc-memos} for short. In this section, we
3089 will learn what they are for, and how they look like --- and also a bit on how
3090 they are produced, even if the details on that will have to wait until
3091 section~\ref{sec:memoization-drivers}.
3092
3093 We will see that when Memoize utilizes memos, c-memos are processed first. But
3094 conceptually, cc-memos are more important, so we will start the discussion with
3095 these.
3096
3097 \subsubsection{Cc-memos (and extern inclusion)}
3098 \label{sec:cc-memos}
3099
3100 When it is input, a cc-memo replicates the effect of the memoized code. This
3101 includes the reproduction of its visual output, which takes the form of
3102 inclusion of any externs produced by memoization. And yes, you got the
3103 implication right: a cc-memo can have any number of associated externs,
3104 including zero, even if the most common case is that of exactly one extern per
3105 cc-memo. The number of externs mostly depends on the memoization driver (see
3106 section~\ref{sec:memoization-drivers}); the default driver always produces
3107 exactly one extern.
3108
3109 A cc-memo is located in the directory determined by key \refmmz{prefix}
3110 (everything up to the final |/| in the value of this key, or the current
3111 directory if this value contains no |/|). You can recognize it by its
3112 filename, which has the following form (\meta{prefix name} is everything
3113 following the final |/| in the value of \refmmz{prefix}):
3114 \begin{center}
3115 \code{\meta{prefix name}\meta{code md5sum}-\meta{context md5sum}.memo}
3116 \end{center}
3117
3118 In fact, this is how Memoize recognizes --- or rather, searches for --- a
3119 cc-memo as well: Memoize will utilize a cc-memo when the code and the context
3120 MD5 sum computed during an attempted utilization match the code and the context
3121 MD5 sum computed during some previous memoization (for details on the context
3122 MD5 sum, see section~\ref{sec:c-memos}). In detail, a cc-memo is created at
3123 the end of memoization, at which point Memoize computes the MD5 sum of the
3124 memoized code and the MD5 sum of the context, and writes the results of
3125 memoization into the cc-memo identified by (the prefix and) these two MD5 sums.
3126 And when Memoize, on a subsequent compilation, encounters a piece of memoized
3127 code, it again computes the MD5 sum of that code and the MD5 sum of the
3128 context, and tries to input the cc-memo identified by (the prefix and) these
3129 two MD5 sums. If the inputting is successful, we have utilized the cc-memo
3130 (which in the typical case amounts to including the one associated extern); if
3131 the cc-memo cannot be found, Memoize starts the memoization process, which
3132 creates the memos and the externs.
3133
3134 Let us take a look at the contents of a cc-memo in detail. Here's a typical
3135 cc-memo (it belongs to the titlepage penguin):
3136
3137 \begingroup
3138 \input{\exampledir titlepage.ccmemo}% --> \mymemo
3139 \tcbinputexample{titlepage}[.ccmemo]{
3140 listing only,
3141 no attachment,
3142 example title=\mymemo,
3143 }
3144 \endgroup
3145
3146 A cc-memo begins by listing the externs which the memo will (actually, might)
3147 attempt to include into the document. When the cc-memo is input, each
3148 \refcmd{mmzResource} command checks if the given extern exists. If some
3149 existence check fails, Memoize enters the memoization mode, same as if the
3150 cc-memo itself did not exist. If all the resources pass the existence check,
3151 Memoize inputs the core of the cc-memo, i.e.\ everything following the
3152 \refcmd{mmzMemo} marker.
3153
3154 The core might contain arbitrary code, but most often, it will consist of only
3155 two commands. The first one is \refcmd{quitvmode} and it is included if the
3156 extern was \refmmz{capture}d into a horizontal box (which is the usual
3157 situation). The second one is \refcmd{mmzIncludeExtern}, and it is this
3158 command which actually includes the extern into the document upon inputting the
3159 cc-memo. (The third command above, \refcmd{mmzStepPgfPictureId}, is written
3160 by PGF support, see section~\ref{sec:support:pgf}.) The core code is executed
3161 without introducing any groups, i.e.\ the effect of any local assignments in
3162 the cc-memo will persist into the code following the memoized code.
3163
3164 Command \refcmd{mmzIncludeExtern} takes nine parameters. The first is the
3165 sequential number of the extern associated with the cc-memo, starting with 0;
3166 usually, this is simply 0 as most memos are associated with a single extern.
3167 The second one is a \refcmd{hbox} or \refcmd{vbox}, noting the type of the box
3168 the memoized code was externalized into. The next three numbers are the
3169 expected width, height and the depth of the extern. Finally, we have the four
3170 \refmmz{padding} amounts (left, bottom, right and top). We should arrive at
3171 the expected size after trimming the extern PDF by the
3172 padding amounts; Memoize will complain if we don't.
3173
3174 Let's look at a more interesting cc-memo. Using the advising framework,
3175 described in section~\ref{sec:tut:automemoization-details}, Memoize hacks
3176 \refcmd{label} to support \cs{label}s inside memoized code --- the following
3177 code ``just works.''
3178
3179 \tcbinputexample{label}(.c1){
3180 comment={\centering
3181 \includeexamplepdf[document page, after title pre={\ (compilation 1)},
3182 left=1.5mm, right=1mm]{page=2}\quad
3183 \includeexamplepdf[document page, after title pre={\ (compilation 2)},
3184 left=1.5mm, right=1mm](.c2){page=1}\quad
3185 \includeexamplepdf[document page, after title pre={\ (compilation 3)},
3186 left=1.5mm, right=1mm](.c3){page=1}
3187 },
3188 }
3189
3190 Everything seems normal --- after the first compilation, we get ``\textbf{??}''
3191 because the label has not made it into the |.aux| file yet, but in subsequent
3192 compilations, we learn where the penguin lives --- but it is far from normal
3193 under the hood. If we de-hacked \cs{label} by writing
3194 \refcmd{mmzset}\bracestt{\refmmz{deactivate}=\cs{label}}, the third compilation
3195 (and subsequent compilations) would revert to ``\textbf{??}''. Why would that
3196 happen? The memoized code containing the \cs{label}s is only executed in the
3197 first compilation; in the subsequent compilations, we're simply inputting the
3198 cc-memo, so the memoized code, including any \cs{label}s in contains, is not
3199 compiled, and the labels don't get into the |.aux| file anymore.
3200
3201 The \cs{label} hack deploys Memoize's ability to put arbitrary code into the
3202 cc-memo. During memoization, the memoized code may add arbitrary code to
3203 register \refcmd{mmzCCMemo}, and the contents of this register at the end of
3204 the memoization form the free-form part of the cc-memo.\footnote{This is also
3205 how the above-described code containing \refcmd{mmzIncludeExtern} gets into
3206 the cc-memo. The code is produced by \refcmd{mmzExternalizeBox} and appended
3207 to \refcmd{mmzCCMemo} by the default memoization driver
3208 \refcmd{mmzSingleExternDriver}; see section~\ref{sec:memoization-drivers} for
3209 details.} When the hacked \cs{label} is encountered during memoization, it
3210 appends \refcmd{mmzLabel}\marg{label name}\marg{current label value} to
3211 \refcmd{mmzCCMemo}, so this command winds up in the cc-memo. It is then a
3212 simple job for \refcmd{mmzLabel}, executed when the cc-memo is input at
3213 subsequent compilations, to temporarily store \meta{current label value} (i.e.\
3214 the contents of \refcmd{@currentlabel} at the time the \cs{label} was invoked) back
3215 into \cs{@currentlabel} and to execute \cs{label}\marg{label name}. In effect, any
3216 \cs{label} command contained within the memoized code is executed at every
3217 compilation, even if the memoized code itself is not compiled.
3218
3219 \begingroup
3220 \input{\exampledir label.ccmemo}
3221 \tcbinputexample{label}[.ccmemo]{float,
3222 listing only, right=1.5mm,
3223 example title=\mymemo,
3224 no attachment,
3225 }
3226 \endgroup
3227
3228 We will continue the discussion of \refcmd{label} in section~\ref{sec:label+} using
3229 a funkier example.
3230
3231
3232 \subsubsection{C-memos (and context)}
3233 \label{sec:c-memos}
3234
3235 As explained in the previous section, a cc-memo belonging to a piece of
3236 memoized code is identified by two MD5 sums: the MD5 sum of the memoized code,
3237 and the MD5 sum of the associated context. However, when Memoize encounters
3238 some code submitted to memoization, the context expression is not yet fully
3239 known, as it may be adjusted by the memoized code itself during memoization ---
3240 and this potential adjustment is crucial for \cs{ref} and friends to work as
3241 advertised (see section~\ref{sec:cross-referencing}). Upon being invoked,
3242 Memoize therefore cannot immediately attempt to input the cc-memo; it needs to
3243 first learn about the context adjustments. Here's where c-memos enter the
3244 picture: \emph{the primary job of a c-memo is to store the context adjustments
3245 made by the memoized code}. Let's see how this works in detail.
3246
3247 Same as cc-memos, c-memos are located in the directory determined by key
3248 \refmmz{prefix}, and their filenames start by \meta{prefix name} determined by
3249 the same key. However, a c-memo belonging to some memoized code is identified
3250 by the MD5 sum of that code alone:
3251 \begin{center}
3252 \code{\meta{prefix name}\meta{code md5sum}.memo}
3253 \end{center}
3254
3255 The c-memo is created at the end of the memoization process. At that time, the
3256 context expression is fully known, as the memoized code was already processed.
3257 Even more, Memoize keeps track of both the state of the context expression
3258 prior to memoization, stored in token register \refcmd{mmzContext}, and of the
3259 \emph{additions} to the context expression made by the memoized code, which are
3260 stored in token register \refcmd{mmzContextExtra}. (Incidentally, key
3261 \refmmz{context} automatically adapts to the situation by appending to
3262 \refcmd{mmzContext} outside memoization and to \refcmd{mmzContextExtra} during
3263 memoization.) The complete context expression is the concatenation of the
3264 contents of these two registers, but it is only the context expression
3265 additions, i.e.\ the contents of \refcmd{mmzContextExtra}, which Memoize stores
3266 into the c-memo, with the idea that during subsequent compilations, the initial
3267 context (\refcmd{mmzContext}) will be set up again via ``normal'' compilation,
3268 while inputting the c-memo will restore the additions, jointly reconstructing
3269 the complete context expression associated with a piece of memoized code to
3270 what it was at the end of memoization.
3271
3272 We can now complete the picture of a utilization attempt started in
3273 section~\ref{sec:cc-memos}. Memoize begins by trying to input the c-memo; this
3274 can be done as the c-memo can be identified based solely on (the MD5 sum of)
3275 the memoized code. If the c-memo does not exist, Memoize starts the
3276 memoization process, which will produce the memos and the externs. But if it
3277 does exist, inputting it reconstructs the context expression to the state at
3278 the end of memoization. Therefore, as the MD5 sum of the \emph{expansion} of
3279 the context expression \emph{at the end of memoization} is baked into the
3280 cc-memo filename, trying to load the cc-memo identified by (the prefix, the
3281 code MD5 sum and) the MD5 sum of the \emph{expansion} of the context expression
3282 \emph{at attempted utilization} will succeed precisely when the context
3283 remained unchanged from memoization to attempted utilization.
3284
3285 All this might have sounded very complicated, but in the end, most c-memos are
3286 quite boring, the titlepage penguin's c-memo shown below being no exception. A
3287 c-memo starts with the \refcmd{mmzMemo} marker, which is always followed by a
3288 (global) assignment to token register \refcmd{mmzContextExtra}, holding the
3289 context expression additions. As promised, the c-memo below is boring: it
3290 assigns an empty token list to this register, leaving the context expression
3291 as-is. Next comes the free-form part of the memo. Below, it is boringly empty
3292 as well (just the percent sign), but in principle, it will contain any code
3293 gathered in register \refcmd{mmzCMemo} during memoization; see
3294 \ref{sec:per-overlay} for an example. A c-memo is concluded by an optional
3295 part consisting of the \refcmd{mmzSource} marker, followed by the memoized
3296 code. The source code section is not used by Memoize in any way and can be
3297 switched off by \code{\refmmz{include source in cmemo}=false}; it is included
3298 by default so that an interested user can know which code produced which memo,
3299 which can be useful if one wants to trigger recompilation of an extern by
3300 deleting the corresponding memo. Incidentally, any newlines in the source code
3301 are lost in the c-memo replica (unless \refmmz{verbatim} is in effect), but we
3302 will only see this once we arrive at the \pkg{beamer} example below.
3303
3304 \begingroup
3305 \input{\exampledir titlepage.cmemo}
3306 \tcbinputexample{titlepage}{% todo
3307 listing only,
3308 listing file=\exampledir\mymemo,
3309 example title=\mymemo,
3310 }
3311 \endgroup
3312
3313 A c-memo of code containing a cross-reference will prove more interesting. The
3314 following c-memo was produced by the |ref| example from section
3315 \ref{sec:cross-referencing}. As we know from that section, when a \refcmd{ref} (as
3316 hacked by Memoize's \refmmzauto{ref} key) occurs in some memoized code, it
3317 appends the cross-reference to the context. In the c-memo below, this is
3318 reflected by the (global) assignment of an expression containing the
3319 cross-reference macro to token register \refcmd{mmzContextExtra}, holding
3320 the context expression additions.
3321
3322 \begingroup
3323 \input{\exampledir ref.cmemo}
3324 \tcbinputexample{ref}[.cmemo]{
3325 listing only,
3326 no attachment,
3327 example title=\mymemo,
3328 }
3329 \endgroup
3330
3331
3332 \subsubsection{More on \texorpdfstring{\cs{label}}{\textbackslash label}}
3333 \label{sec:label+}
3334
3335 A \refcmd{label} inside memoized code works out of the box in the usual situation
3336 when label value is fully determined by the memoized code, as in the example in
3337 section~\ref{sec:cc-memos}, where the memoized code contained the outermost
3338 (and only) \env{enumerate} environment. However, the out of the box approach does
3339 not work if the label value is (fully or partially) determined outside the
3340 memoized code. To illustrate the problem, and some potential solutions, we
3341 define two very simple enumeration environments, |listi| and |listii|, which
3342 use counters |counti| and |countii|, and which are intended as the outer and
3343 the inner environment, respectively. Our interest here is in the inner
3344 environment, |listii|. While it prefixes each item by an indented
3345 |\thecountii)|, the label is a composite of both counters:
3346 |\thecounti\thecountii|. The label is stored into \refcmd{@currentlabel}, so
3347 referencing works as usual. However, problems arise when we automemoize the
3348 inner environment.
3349
3350 \tcbinputexample{label+}(.c1){
3351 sidebyside, lefthand ratio=0.53,
3352 comment={\centering\includeexamplepdf[document page](.c2){page=1}},
3353 }
3354
3355 While the result looks fine at first, changing the order of |listii|
3356 environments, for example by moving ``pets'' below ``domestic'', will result in a
3357 problem: the reference at the bottom will remain unchanged. This is so because
3358 the reference text is baked into the cc-memo, as shown below.
3359
3360 \begingroup
3361 \input{\exampledir label+.ccmemo}
3362 \tcbinputexample{label+}[.ccmemo]{
3363 listing only,
3364 no attachment,
3365 example title=\mymemo,
3366 }
3367 \endgroup
3368
3369 How can we remedy this? The manual option is to force the recompilation of the
3370 extern by putting an (invisible) reference to the outer item into the inner
3371 item: add |\label{item:pets}| to item ``pets'' and refer to it at ``dog'' by
3372 \refcmd{mmzNoRef}|{item:pets}|.%
3373 \attachexample[label+mmzNoRef.tex][\exampledir label+.tex.c3.attachment]
3374
3375 An automatic variant of the recompilation solution is to add \cs{@currentlabel}
3376 to the context upon memoizing |listii|. This can be achieved by adding
3377 \refmmz{context}|={@currentlabel={\csuse{@currentlabel}}}| to the \refmmz{auto}
3378 declaration for |listii|. The downside of this approach is that every |listii|
3379 will get reexternalized upon movement, whether it actually contains a label or
3380 not.\attachexample[label+context.tex][\exampledir label+.tex.c4.attachment]
3381
3382 In fact, given that the externs produced by the inner environment do not
3383 contain the value of the outer counter, it seems wasteful to recompile any
3384 extern just to change the reference. And indeed, it is possible to avoid this,
3385 but the approach unfortunately requires adapting the inner environment code
3386 (and this is why I have not illustrated the problem using an environment of an
3387 elaborate package like \pkg{enumitem}). The idea is to ``unbake'' the
3388 reference to the outer item in the cc-memo. We can achieve this by changing
3389 |listii| to define \cs{@currentlabel} to be |\unexpanded{\thecounti}\thecountii|.
3390 Under this definition, the cc-memo will contain |\mmzLabel
3391 {item:dog}{\thecounti a}|, and rearranging |listii| environments will produce
3392 (upon two compilations, of course) the correct reference without recompiling
3393 the extern. Note again, however, that this solution can only work when the
3394 value of the outer counter does not appear in the extern, i.e.\ it would not
3395 work the ``dog'' item was prefixed by |1a)| rather than simply |a)|. In those
3396 cases, one should deploy one of the other solutions.%
3397 \attachexample[label+listii.tex][\exampledir label+.tex.c5.attachment]
3398
3399 The final solution, presented below, is an elaboration on the second one.
3400 Rather than append \cs{@currentlabel} to the context immediately upon beginning
3401 to memoize environment |listii|, we will at that point redefine \cs{label} to do
3402 that. In effect, changing the location of |listii| will only recompile it if
3403 it contains a \cs{label}.
3404
3405 As announced, we redefine \cs{label} once the memoization of |listii| begins,
3406 so within \refmmz{at begin memoization}.\footnote{Another generally good
3407 location for such redefinitions is among the auto-options of |listii|. We
3408 could include an \refmmz{auto}\cs{label}\marg{...} there, or a |/utils/exec|
3409 with \refcmd{AdviceSetup}. However, in this particular case this would be
3410 wasteful, as it would be applied regardless of whether memoization will take
3411 place or not, whereas we only need the redefined \cs{label} when memoizing.}
3412 However, we do not redefine \cs{label} directly, as Memoize advises this
3413 control sequence out of the box (see
3414 section~\ref{sec:tut:automemoization-details} for details). What we redefine
3415 is the command which the advising framework executes instead of \cs{label} ---
3416 its so-called \refmmzauto{outer handler} --- and we do this by calling
3417 \refcmd{AdviceSetup}, the low-level variant of the familiar key
3418 \refmmz{auto}.\footnote{We could have also used \refmmz{auto}, but we don't,
3419 because (a) \refcmd{AdviceSetup} is faster, (b) it is easier to prepend
3420 material to a handler using the low-level interface, and (c) I wanted to
3421 showcase \refcmd{AdviceSetup}.} The first argument of \refcmd{AdviceSetup}
3422 is the installation path (|/mmz|), the second one the command or environment we
3423 are submitting to the framework (\cs{label}), and the third one the setup
3424 \emph{code} --- here lies the biggest difference between \refmmz{auto} and
3425 \refcmd{AdviceSetup}: the former expects a keylist, and the latter \hologo{TeX}
3426 code which directly manipulates settings macros like
3427 \refcmd{AdviceOuterHandler} (for the full list, see
3428 section~\ref{sec:tut:automemoization-details} or~\ref{sec:ref:advice}).
3429
3430 \tcbinputexample{label+}(.c6){listing only, float,
3431 attachment=label+auto.tex,
3432 }
3433
3434 Within \refcmd{AdviceSetup}, we prefix (using macro \cs{preto} of package
3435 \pkg{etoolbox}) the original outer handler \refcmd{AdviceOuterHandler} by code
3436 which causes \cs{outerlabeltocontext} to be executed at the end of memoization
3437 (by globally appending this macro to \refcmd{mmzAtEndMemoizationExtra};
3438 ``Extra'' because we're appending during memoization). It is
3439 \cs{outerlabeltocontext} which then appends \refcmd{@currentlabel} to the
3440 context (\refcmd{mmzContextExtra}; again, ``Extra'' because we're appending
3441 during memoization),\footnote{As a courtesy, we clear out macro
3442 \cs{outerlabeltocontext} once it did it's job, so that multiple \cs{label}s
3443 do not include multiple \cs{@currentlabel}s into the context. But the code
3444 would work even without this addendum, can you see why?} and it is crucial
3445 that this happens at the end of memoization rather than when \cs{label} is
3446 executed. When \cs{label} is executed, we're inside an inner item, and
3447 \cs{@currentlabel} refers to that item, while at the end of memoization, the
3448 value of this macro equals the value at the beginning of memoization, namely
3449 the label of the outer item. While the outer list remains unshuffled, the
3450 value of \refcmd{@currentlabel} that contributes to the context MD5 sum during
3451 the utilization attempt will therefore match the value which contributed to the
3452 context MD5 sum during memoization, resulting in matching MD5 sums and
3453 therefore in actual utilization of the extern; once the outer list is shuffled,
3454 this will cease to be the case and the extern will be recompiled.
3455
3456
3457 \subsubsection{The Beamer support explained}
3458 \label{sec:per-overlay}
3459
3460 The implementation of \refmmz{per overlay}, which makes memoization sensitive
3461 to Beamer overlays, provides an example of a complex interaction between
3462 various components of memoization. At the core, the Beamer support works by
3463 adjusting the context, but we will also have the occasion to observe the
3464 free-form part of the c-memo, add a bit of extra code to the cc-memo, and
3465 deploy several memoization hooks. We will show the complete Beamer support
3466 code later on; let us build our understanding of that code step by step.
3467 (Before you read on, you might want to refresh your memory about the \pkg{beamer}
3468 example from section~\ref{sec:tut:beamer}, as we will refer to it in the
3469 present section.)
3470
3471 The core idea behind \refmmz{per overlay} is to append the current beamer
3472 overlay number to the context:\cprotect\footnote{As we saw in
3473 section~\ref{sec:tut:beamer}, it is convenient to execute \refmmz{per
3474 overlay} inside memoized code. But remember, from
3475 section~\ref{sec:cross-referencing}, that when \refmmz{context} is executed
3476 from within the memoized code, its argument winds up in the c-memo. As the
3477 c-memo is processed under the normal category code regime, where |@| is not a
3478 letter, we have to access |\beamer@overlaynumber| using the |\csname
3479 ... \endcsname| construct.} \refmmz{context}|={overlay=\csname
3480 beamer@overlaynumber\endcsname}|. This makes Memoize produce a separate
3481 extern for each overlay. However, only the first of these externs will get
3482 utilized on subsequent compilations, in general at least. Even worse, we will
3483 lose each frame whose creation is driven solely by the memoized code. We will
3484 lose the second overlay in our example (i.e.\ in the \pkg{beamer} example from
3485 section~\ref{sec:tut:beamer}) as the second overlay was only created because
3486 Beamer encountered |only={2}{...}| (resolving to |\only<2>{...}| under the
3487 hood) inside the picture code; once we utilize the extern instead of compiling
3488 the picture on the first overlay, the |\only| command is not executed anymore,
3489 so Beamer thinks it is done with the frame.
3490
3491 As the compilation of our picture is substituted by utilization of its
3492 cc-memos, we have to somehow drive the creation of the necessary overlays from
3493 these files. An easy way to achieve this is to furnish them with a dummy
3494 |\only<|\meta{final overlay number}|>{}|,\footnote{`The final overlay' here
3495 should be understood as relative to our memoized picture, i.e.\ as the final
3496 overlay containing the memoized picture.}\footnote{Actually, putting this
3497 \cs{only} command only into the first cc-memo would suffice, but would be
3498 harder to implement.}\footnote{In the actual implementation, the \meta{final
3499 overlay number} is also prefixed by \code{all:}, to have this work in all
3500 Beamer modes.} but there is a problem: the final overlay number is unknown
3501 when we're memoizing our picture --- it is unknown even when we're memoizing
3502 the picture on final overlay itself (we simply don't know yet that this overlay
3503 will end up being the final one), let alone during the memoization on the first
3504 overlay.
3505
3506 The solution exploits the fact that \emph{the c-memo is rewritten at each
3507 memoization}: at each memoization of our picture, we store the the
3508 \emph{current} overlay number to the c-memo; after all memoizations, the c-memo
3509 will thus contain the number of the \emph{final} overlay containing our
3510 memoized picture. To access this number from the cc-memo, we store it as a
3511 macro definition, and then use the defined macro, |\mmzBeamerOverlays|, in the
3512 overlay specification of the dummy |\only|. Below, you can see all this in
3513 code, as the argument to \refmmz{at begin memoization}.
3514
3515 \tcbinputexample{per-overlay-v1}[.sty]{
3516 listing only,
3517 title={The implementation of \refmmz{per overlay} (first attempt)},
3518 }
3519
3520 A couple of remarks are in order here. First, the definition of
3521 |\mmzBeamerOverlays| in the c-memo is global, because it will be accessed from
3522 the cc-memo, but the cc-memo is input after closing the memoize group (which
3523 the c-memo \emph{is} processed in). Second, using \refmmz{at begin
3524 memoization} makes it possible to use \refmmz{per overlay} both outside and
3525 during memoization: if \refmmz{at begin memoization} is executed outside
3526 memoization, its argument is (locally) stored into hook
3527 \refcmd{mmzAtBeginMemoization}, to be called at the beginning of each
3528 memoization; if the key executed outside memoization, the argument is executed
3529 immediately. Third, the Memoize keys in the definition of \refmmz{per overlay}
3530 are prefixed with \code{\refkeypath{/mmz}/}, so that this key can be called from
3531 \pkg{pgfkeys} option lists of other packages, for example the option list of
3532 the \env{tikzpicture} environment, as shown in the example in
3533 section~\ref{sec:tut:beamer}.
3534
3535 I used the above version of \refmmz{per overlay} for quite a while. In
3536 general, it worked as I expected, but there were glitches. Occasionally, the
3537 picture would appear on the wrong overlay, or I would get an extra overlay, or
3538 perhaps lose an overlay. Eventually, I figured out this happens when I play
3539 with the overlay structure of the frame: when I add or remove a \cs{pause} or
3540 similar. In hindsight, it is easy to see what was happening. Once the picture
3541 is memoized, it is fixed, forever, which extern will appear on which overlay.
3542 I cannot expect the extern--overlay correlation to change just because I added
3543 a \cs{pause} in front of the picture. Furthermore, the number of overlays the
3544 memos will drive to be created is fixed as well. If I memoize the picture
3545 while it follows a \cs{pause}, and the picture creates 10 overlays, the c-memo
3546 will define |\mmzBeamerOverlays| to 11. So what, if I then remove that
3547 \cs{pause}! The c-memo will still define |\mmzBeamerOverlays| to 11, and drive
3548 the creation of 11 overlays --- one too many.
3549
3550 By now, the road ahead is probably clear --- we put the |beamerpauses| counter
3551 into the context --- but we will see there are still obstacles on the way. The
3552 issue is that the context is evaluated at the \emph{end} of memoization (so
3553 that those cross-references from section~\ref{sec:cross-referencing} actually
3554 get into it). However, the memoized code might contain a \cs{pause} or similar
3555 itself, and change the value of |beamerpauses|. For one, this means that we
3556 have to write down the changed value of |beamerpauses| into the cc-memo; below,
3557 we do this using key \refmmz{at end memoization} (the code given code to this
3558 key is executed after the driver but before Memoize writes down the memos and
3559 ships out the extern pages; the key itself may be executed either before or
3560 during memoization). Furthermore, if the memoized code changes the value of
3561 |beamerpauses|, the value of |beamerpauses| at the attempted utilization, which
3562 would nicely match the value from the start of memoization, will never match
3563 the changed, final value from the end of memoization, in effect preventing our
3564 hard-won externs from ever getting utilized.
3565
3566 We therefore have to invent a way to get the memoization-\emph{initial} value
3567 of |beamerpauses| into the context.\footnote{This imposes a requirement on the
3568 in-code usage of \refmmz{per overlay}, namely, that it should be executed
3569 prior to any changes of |beamerpauses|.} Fine, we store it in a
3570 macro,\footnote{We must define \cs{mmzBeamerPauses} globally, because
3571 \refmmz{per overlay} can be arbitrarily deeply embedded in the memoized
3572 code.} |\mmzBeamerPauses|, when we start the memoization (\refmmz{at begin
3573 memoization}), and put |\mmzBeamerPauses| rather than |beamerpauses| into the
3574 context. Will this work? Not yet, because |\mmzBeamerPauses| is undefined at
3575 utilization. We need to set up the context expression so that it will expand
3576 to the value of |\mmzBeamerPauses| at (the end of) memoization, and to the
3577 value of |beamerpauses| at utilization. This leads to the
3578 |pauses=\ifmemoizing...| part of the context expression in (the final version
3579 of) the Beamer support code below.
3580
3581 \medskip % manual, to avoid an orphan
3582
3583 \tcbinputexample{per-overlay}[.excerpt]{%
3584 listing only, no .listing,
3585 title={The implementation of \refmmz{per overlay}},
3586 }
3587
3588 Are we done? Almost. The final issue is that once we have introduced support
3589 for pauses, we have to relativize |\mmzBeamerOverlays| (the final overlay
3590 number) to |beamerpause|. So instead of a simple |\gdef\mmzBeamerOverlays| in
3591 the first version, we define
3592 \docaux{cmd}{mmzSetBeamerOverlays}\nohyphen\marg{beamer pauses}\marg{final
3593 overlay number}, which sets |\mmzBeamerOverlays| only if \meta{beamer pauses}
3594 argument matches the value of |beamerpauses| (at its invocation in the c-memo).
3595 Well, the macro has some other housekeeping to do as well: it is
3596 self-replicating, so that during potential memoization, the
3597 |\mmzBeamerOverlays| values belonging to non-current |beamerpauses| values get
3598 rewritten into the c-memo.\footnote{As replication should only occur during
3599 memoization (actually, it \emph{can} only occur then, anyway), the
3600 instruction to append to the c-memo is appended to the
3601 \cs{mmzAtBeginMemoization} hook (the low-level interface to \refmmz{at begin
3602 memoization}). Note that the assignment to this hook must be local (once
3603 local, always local), and it \emph{can} be local because of a little
3604 implementation detail: while the c-memo is processed in the memoize
3605 \hologo{TeX} group, we don't open an additional group to process it; so the
3606 local effects from c-memo will persist into memoization (but not into
3607 utilization, because remember that the memoize group is closed before
3608 inputting the cc-memo).} (Fine, there is another fine detail, regarding
3609 anti-pollution: the macro also ensures that, relative to \marg{beamer pauses},
3610 only the instance with the greatest \meta{final overlay number} is replicated.)
3611
3612 We are now truly done, and we can look at the final result, the c-memo and the
3613 cc-memo belonging to the extern on the first overlay of the example from
3614 section~\ref{sec:tut:beamer}. Specifically, look at the
3615 \code{\refcmd{mmzSetBeamerOverlays}\braces{1}\braces{2}}, which says that the
3616 extern chain started when |beamerpauses| equals 1 should continue up to overlay
3617 2, and at the (expanded) context included at the end of the cc-memo, courtesy
3618 of \refmmz{include context in ccmemo}, where you can see that the cc-memo will
3619 be used when on the first overlay (|overlay=1|) when preceded by no \cs{pause}
3620 command (|pauses=1|).
3621
3622 \begingroup
3623 \input{\exampledir beamer.cmemo}
3624 \tcbinputexample{beamer}[.cmemo]{
3625 listing only,
3626 no attachment,
3627 example title=\mymemo,
3628 }
3629 \input{\exampledir beamer.ccmemo}
3630 \tcbinputexample{beamer}[.ccmemo]{
3631 listing only,
3632 no attachment,
3633 example title=\mymemo,
3634 }
3635 \endgroup
3636
3637
3638 \paragraph{An application of \refmmz{salt}}
3639
3640 Knowing the role of a c-memo in driving the creation of Beamer overlays, we are
3641 ready for an application of a key from the \refmmz{salt} family. Key
3642 \refmmz{salt} has a similar effect to \refmmz{context}: it contributes towards
3643 an md5 sum of a memo. However, unlike \refmmz{context}, \refmmz{salt} impacts
3644 the code hash, which is actually the md5 sum of the concatenation of the salt
3645 and the memoized code. (As this md5 sum is calculated prior to memoization,
3646 using \refmmz{salt} and friends during memoization has no effect.)
3647
3648 While \refmmz{salt} and \refmmz{context} are most often interchangeable, the
3649 following example crucially requires the former. The goal is to have the memos
3650 and externs depend on the definition of command \cs{N}. In non-Beamer
3651 documents, we can achieve this by putting the meaning of \cs{N} into the
3652 context. In Beamer documents, however, this does not work. Each meaning of
3653 \cs{N} should form a separate chain of externs, and these chains may be of
3654 different length (and in the example, they are). As we are storing the length
3655 of the chains in c-memos (in \refcmd{mmzSetBeamerOverlays}), it follows that
3656 each value of \cs{N} should receive its own c-memo. A c-memo is identified by
3657 a single md5 sum, the code md5 sum, so the meaning of \cs{N} must contribute
3658 towards this md5 sum. In other words, we must salt the memoized code with the
3659 meaning of \cs{N}. Below, if we had put \cs{N} into the context rather than
3660 the salt, the two frames would create the same number of overlays: two, the
3661 number of overlays created by the final frame.
3662
3663
3664 \tcbinputexample{salt}{
3665 sidebyside, lefthand ratio=0.65,
3666 comment={\centering
3667 \includeexamplepdf[extern page,left=2mm,right=2mm]{page=1,trim=1in 1in 1in 1in}\\[1ex]
3668 \includeexamplepdf[extern page,left=2mm,right=2mm]{page=3,trim=1in 1in 1in 1in}\\[1ex]
3669 \includeexamplepdf[extern page,left=2mm,right=2mm]{page=5,trim=1in 1in 1in 1in}\\[1ex]
3670 \includeexamplepdf[extern page,left=2mm,right=2mm]{page=7,trim=1in 1in 1in 1in}\\[1ex]
3671 \includeexamplepdf[extern page,left=2mm,right=2mm]{page=9,trim=1in 1in 1in 1in}
3672 },
3673 }
3674
3675
3676 \subsection{Record files}
3677 \label{sec:record-files}
3678
3679 We have seen that externalization is a two-step process in Memoize: as it is
3680 impossible for \hologo{TeX} to create multiple PDFs during a single
3681 compilation, the externs are first dumped into the document PDF as special
3682 extern pages, and only later extracted from the main document into separate PDF
3683 files. But extraction requires a complete PDF, which is unavailable even at
3684 the very end of the compilation which produces the externs. The externs can
3685 therefore only be extracted \emph{after} that compilation (either before or at
3686 the beginning of the next one), and this necessitates some form of
3687 communication whereby the memoization step informs the extraction step which
3688 pages should be extracted from the document PDF and into which (PDF) files they
3689 should be stored. This communication is implemented through auxiliary files
3690 called \emph{record files}.
3691
3692 \subsubsection{The \texttt{.mmz} file}
3693 \label{sec:.mmz}
3694
3695 By default, Memoize records the information needed for the extraction in a file
3696 named \meta{document name}\dmmz,\footnote{For \hologo{TeX}perts: the
3697 \meta{document name} is of course the expansion of \cs{jobname}.} henceforth
3698 a \dmmz file. In fact, this file contains more than information on externs
3699 created during the last compilation: it records which memos and externs were
3700 either used or created during the compilation. The full information contained
3701 in the \dmmz file is used by the clean-up script \refscript{memoize-clean.pl}
3702 to safely remove stale memos and externs. Let us take a look at the \dmmz file
3703 produced by the titlepage illustration. In fact, we have two versions of this
3704 file, as it changes upon the second compilation.
3705
3706 \tcbinputexample{titlepage}[.mmz-pre]{
3707 listing only, no .listing, no attachment, float,
3708 after title pre={\ (after the first compilation)},
3709 listing options app={breakatwhitespace=false,prebreak=\coloredpercentchar},
3710 }
3711 \tcbinputexample{titlepage}[.mmz-post]{
3712 listing only, no .listing, no attachment, float,
3713 after title pre={\ (after subsequent compilations)},
3714 listing options app={breakatwhitespace=false,prebreak=\coloredpercentchar},
3715 }
3716
3717 As you can see, the \dmmz file takes the form of a \hologo{TeX} script (the
3718 format was chosen because it facilitated the implementation of the internally
3719 triggered \hologo{TeX}-based extraction). The crucial lines in this file, and
3720 the only lines used by the extraction script, occur in the first version of the
3721 file: they contain command \refcmd{mmzNewExtern}, which informs the extraction
3722 script that it should extract the document page given by the second argument
3723 into the extern file given by the first argument.\footnote{If you look at the
3724 \dmmz file after extracting the externs using \refscript{memoize-extract.pl}
3725 without the \refscript{memoize-extract.pl--keep} option, you will find that
3726 the \refcmd{mmzNewExtern} commands are commented out; this is to prevent
3727 multiple extractions (even if they are harmless).} (The following two
3728 arguments provide the expected width and height of the extern; the
3729 extraction script may check whether the extern size conforms to these
3730 expectations, but this is not crucial, as the extern size is checked every time
3731 it is included anyway.)
3732
3733 A \dmmz file also contains a record of the memos (both c-memos and cc-memos)
3734 created in the last compilation; this information is provided by the sole
3735 argument of commands \refcmd{mmzNewCMemo} and \refcmd{mmzNewCCMemo}. And once
3736 memos and externs get used in subsequent compilations, the \dmmz file will
3737 reflect this with \refcmd{mmzUsedCMemo}, \refcmd{mmzUsedCCMemo} and
3738 \refcmd{mmzUsedExtern}, as shown in the second version of the file above.
3739
3740 Finally, both versions illustrate that a \dmmz file always begins with command
3741 \refcmd{mmzPrefix} and ends with the |\endinput| marker. The argument of
3742 \refcmd{mmzPrefix} is the prefix to the memo and extern files, as
3743 determined by the invocation of key \refmmz{prefix}. The initial
3744 \refcmd{mmzPrefix} line is written to the \dmmz file at the beginning of the
3745 document, but an additional \refcmd{mmzPrefix} line will occur for every
3746 invocation of \refmmz{prefix} in the document body. Finally, the |\endinput|
3747 marker signals that the \dmmz file is complete.
3748
3749 As mentioned above, the full contingent of \dmmz file commands is only used by
3750 the clean-up script \refscript{memoize-clean.pl}. By default, this script
3751 removes all memos and externs with the prefix given by \refcmd{mmzPrefix}
3752 (relative to the directory hosting the \dmmz file) but those listed by any of
3753 \refcmd{mmzNewCMemo}, \refcmd{mmzNewCCMemo}, \refcmd{mmzNewExtern},
3754 \refcmd{mmzUsedCMemo}, \refcmd{mmzUsedCCMemo} and \refcmd{mmzUsedExtern}.
3755 Furthermore, the clean-up script will cowardly refuse to delete anything if the
3756 \dmmz file does not end with |\endinput|, as this means that the compilation
3757 ended prematurely and that the \dmmz file might not mention all memos and
3758 externs actually used in the document. If given option
3759 \refscript{memoize-clean.pl--all}, the clean-up script removes even the memos
3760 and externs mentioned in the \dmmz file, and as this option is intended to
3761 bring Memoize to a clean slate after any ``disasters,'' the
3762 \refscript{memoize-clean.pl--all} mode also ignores the potential absence of
3763 the |\endinput| marker. Incidentally, the \refscript{memoize-clean.pl--all}
3764 mode is also the raison d'être for \refcmd{mmzPrefix}: while the prefix is
3765 usually recognizable from \refcmd{mmzNewCMemo} and friends, these commands
3766 might not make it into the \dmmz file in a fatally failed compilation, but it
3767 is precisely such compilations that could occasionally require the full
3768 clean-up.
3769
3770 \subsubsection{Defining a new record type}
3771 \label{sec:new-record-file}
3772
3773 The \dmmz file is not the only kind of a record file that can be produced by
3774 Memoize. Out of the box, it can also write down the extraction instructions
3775 into a makefile or a shell script. These are useful on systems which have to
3776 employ the \hologo{TeX}-based extraction but cannot trigger it internally.
3777 Running the \hologo{TeX}-based extraction manually would be painful, as it must
3778 be done on extern-by-extern basis, so Memoize offers to automate the extraction
3779 by a makefile or a shell script; here, the record file is named
3780 \code{memoize-extract.\meta{document name}.\meta{record type}} by default,
3781 where \meta{record type} is either \refmmz{record=makefile},
3782 \refmmz{record=sh} (for shell scripts on Linux), or \refmmz{record=bat} (for
3783 shell scripts on Windows).
3784
3785 To turn on recording of an alternate record type, use key
3786 \refmmz{record}|=|\meta{record type}. Memoize can record any number of files
3787 simultaneously, so saying \code{\refmmz{record}=\refmmz{record=sh}} will
3788 produce the shell script alongside \dmmz (Memoize internally executes
3789 \code{\refmmz{record}=\refmmz{record=mmz}} to start recording the \dmmz file);
3790 this should not be a problem, but if you really want to disable the \dmmz file
3791 production, you can say \refmmz{no record}.
3792
3793 The predefined record types are defined through a generic system open to the
3794 user. To define an additional record type, one needs to define, using
3795 \pkg{pgfkeys}, the relevant hooks of the form \refkeypath{/mmz/record/record
3796 type}|/|\meta{hook}. The following \meta{hook}s can be defined (the hooks
3797 not needed for the record file type may be left undefined):
3798 \begin{itemize}
3799 \item Key \refmmz{record/record type/begin} will be executed at the beginning
3800 of the document; it will receive no argument. Use it to open the record
3801 file.
3802 \item Key \refmmz{record/record type/end} will be executed at the end of the
3803 document; it will receive no argument. Use it to close the record file.
3804 \item Key \refmmz{record/record type/prefix} will be executed at the end of the
3805 document and at every invocation of key \refmmz{prefix} in the document body;
3806 it will receive a single argument, the prefix determined by key
3807 \refmmz{prefix}.
3808 \item Keys \refmmz{record/record type/new cmemo}, \refmmz{record/record
3809 type/used cmemo}, \refmmz{record/record type/new ccmemo} and
3810 \refmmz{record/record type/used ccmemo} will be executed after creating or
3811 inputting a memo; they will receive a single argument, the full path to the
3812 memo.
3813 \item Key \refmmz{record/record type/used extern} will be executed after an
3814 extern was included into the document; it will receive a single argument, the
3815 full path to the extern.
3816 \item Key \refmmz{record/record type/new extern} will be executed after
3817 creating creating an extern, more precisely at the end of memoization, right
3818 after shipping out the extern page. It will receive a single argument, the
3819 full path to the extern, but additionally, Memoize prepares the following
3820 macros:
3821 \begin{itemize}
3822 \item \refcmd{ne:externbasepath} holds the full path to the extern, but (unlike |#1|)
3823 without the |.pdf| suffix;
3824 \item \refcmd{ne:pagenumber} holds the ``physical'' page number of the extern page in
3825 the document (the numbering starts by $1$);
3826 \item \refcmd{ne:expectedwidth} and \refcmd{ne:expectedheight} hold the width and
3827 the height (total height, i.e.\ the sum of \hologo{TeX}'s height and depth)
3828 of the extern page.
3829 \end{itemize}
3830 \end{itemize}
3831
3832 Below, we present two simple examples of a record file. The first type simply
3833 records the names of all memos and externs used or created by Memoize; the
3834 resulting file could be included by |.gitignore| to have
3835 \hreftt{https://git-scm.com}{git} automatically ignore all files produced by
3836 Memoize. The second type lists the new externs, each preceded by its page
3837 number in the |.pdf|; this file could be fed to a custom extern extraction
3838 tool.
3839
3840 \tcbinputexample{record-files}{listing only}
3841
3842 \tcbinputexample{record-extern-pages}{listing only}
3843
3844 Finally, note that (unlike memos and externs) record files are auxiliary files
3845 and may be deleted at any time after the extraction of the externs produced in
3846 the final compilation --- actually, even if these externs were not yet
3847 extracted, deleting the record file(s) will merely force their recompilation.
3848
3849
3850 \subsection{The memoization process}
3851 \label{sec:memoization-drivers}
3852
3853 We now turn to the memoization process itself. The job of memoization is to,
3854 while compiling the given code in a regular fashion, prepare the cc-memo
3855 (which, when it is input, will replicate the effect of the given code),
3856 alongside any externs that the cc-memo will include (these hold the typeset
3857 material to be replicated). Clearly, merely compiling the code cannot have
3858 this effect (unless that code was written specifically to support memoization;
3859 more on this later), and this is why the memoized code is typically wrapped by
3860 a \emph{memoization driver}, which can be set using key \Emph{\refmmz{driver}}.
3861 We'll inspect the default memoization driver, \refcmd{mmzSingleExternDriver},
3862 in the first subsection, and we will learn how to write specialized drivers in
3863 the remaining subsections. But first, let us say some words about a
3864 grouping-related \hologo{TeX}nical detail we need to take care about during
3865 memoization.
3866
3867 During memoization, we have to collect certain information, like build the
3868 contents of the cc-memo. Some of that information might be contributed by the
3869 memoized code itself. For example, a \cs{label} ``adds itself'' to the cc-memo
3870 (by appending to token register \refcmd{mmzCCMemo}); a
3871 \refkey{/tikz/remember picture} aborts memoization (by issuing
3872 \refcmd{mmzAbort}); etc. The issue is that the memoized code might open any
3873 number of \hologo{TeX} groups; we have no idea how deeply embedded the
3874 \cs{label} or \refkey{/tikz/remember picture} might be. Therefore, we have
3875 to collect all the information about the ongoing memoization \emph{globally}:
3876 all assignments to \refcmd{mmzCCMemo} must be global; \refcmd{mmzAbort} sets
3877 the underlying conditional globally; etc. (Clearly, all these global variables
3878 are initialized at the start of memoization.)
3879
3880 This was the easy part. An additional complication arises with some options
3881 which may be set either outside memoization, or during this process. For
3882 example, you can append the font size to the context expression in the preamble
3883 (see section~\ref{sec:cross-referencing}), so that the externs will be
3884 automatically recompiled when the font size changes, and clearly, this context
3885 adjustment should respect \hologo{TeX} grouping; but a \cs{ref} or some other
3886 cross-referencing command in the memoized code needs to append to the context
3887 as well, and as this \cs{ref} occurs \emph{within} the memoized code, the
3888 assignment must be global, as explained above.
3889
3890 Mixing the local and global assignments to the token register
3891 \refcmd{mmzContext}, which holds the (in the actual implementation, local)
3892 context expression, will not do. For one, we \emph{do} want to restore the
3893 pre-memoization context expression after we have memoized the code, and
3894 furthermore, mixing local and global assignments to the same variable is not
3895 recommended for save stack reasons anyway.
3896
3897 Memoize addresses this issue by having \emph{two} context registers,
3898 \refcmd{mmzContext} and \refcmd{mmzContextExtra} --- when computing the context
3899 MD5 sum (which happens at the end of memoization), the two registers are
3900 concatenated (the local one comes first). A package writer should know when to
3901 use which register, and how. Outside memoization, one should assign to
3902 \refcmd{mmzContext} --- \emph{locally}. During memoization, one should assign
3903 to \refcmd{mmzContextExtra} --- \emph{globally}. The user interface key
3904 \refmmz{context} respects this requirement automatically: it locally appends to
3905 \refcmd{mmzContext} outside memoization, and it globally appends to
3906 \refcmd{mmzContextExtra} during memoization. (The same idea is applied to the
3907 post-memoization hooks \refmmz{at end memoization} and \refmmz{after
3908 memoization}.)
3909
3910
3911 \subsubsection{The default memoization driver}
3912 \label{sec:memoization-driver-default}
3913
3914 The default memoization driver, \refcmd{mmzSingleExternDriver}, produces
3915 exactly one extern, which contains whatever is typeset by the code submitted to
3916 memoization. The driver compiles the code into a horizontal or vertical box
3917 depending on the value of key \refmmz{capture}. Let us look at the definition
3918 of the driver line by line:
3919
3920 \tcbinputexample{single-extern-driver}[.excerpt]{%
3921 listing only, no .listing,
3922 listing options app={numbers=left, numberstyle=\tiny, numbersep=0.5em},
3923 title=The default memoization driver,
3924 float, no attachment,
3925 }
3926
3927 \begin{enumerate}
3928 \item Macro \refcmd{mmzSingleExternDriver} (and in fact any memoization
3929 driver) takes a single argument, the code to compile. Memoize will call the
3930 driver with the code given as the second argument to \refcmd{Memoize}, but
3931 wrapped in a macro which re-reads it using \refcmd{scantokens} when
3932 \refmmz{verbatim} is in effect.
3933 \item If we're capturing into a horizontal box
3934 (\refmmz{capture}|=|\refmmz{capture=hbox}), we put \refcmd{quitvmode} into
3935 the cc-memo --- putting it to the very beginning should make sure that any
3936 replicated \cs{label} and \cs{index} commands refer to the correct page.
3937 \item We compile the given code, storing the typeset material into a box
3938 (above, a temporary box called |\mmz@box|). |\mmz@capture| resolves into a
3939 box construction command, depending on the value \refmmz{capture}.
3940 \item Macro \refcmd{mmzExternalizeBox} instructs Memoize to externalize the box
3941 given as its first argument. However, this macro does not directly produce
3942 an extern page or write any instructions into the cc-memo; the road to this
3943 final destination is indirect. \refcmd{mmzExternalizeBox} has two effects.
3944 First, it adds the contents of the given box (above, |\mmz@box|) to an
3945 internal box dedicated to holding all the externs produced in this
3946 memoization (the contents of |\mmz@box| remain as they are) --- it is only at
3947 the end of memoization that the contents of this internal box are shipped off
3948 to extern pages. Second, \refcmd{mmzExternalizeBox} produces the code which
3949 will include the extern into the document on subsequent compilations (this
3950 will be a call to \refcmd{mmzIncludeExtern}, potentially prefixed by
3951 \refcmd{quitvmode}; see section~\ref{sec:cc-memos} for details). This code
3952 is stored into the token register gives as the second argument (above,
3953 |\mmz@temptoks|), and it is the responsibility of the driver to include it
3954 into the cc-memo. (In the interest of full disclosure,
3955 \refcmd{mmzExternalizeBox} also updates the list of externs produced in this
3956 memoization. At the end of memoization, this list is written to the
3957 beginning of the cc-memo, resulting in the \refcmd{mmzResource} lines
3958 preceding the \refcmd{mmzMemo} marker.)
3959 \item The construction of the cc-memo is indirect as well. In the third line
3960 of the definition, we globally append the extern-inclusion code residing in
3961 |\mmz@temptoks| to token register \refcmd{mmzCCMemo}. At the end of
3962 memoization, the contents of \refcmd{mmzCCMemo} are written into the cc-memo,
3963 preceded by the \refcmd{mmzMemo} marker.
3964 \item We put the typeset material into the document, again preceded by
3965 \refcmd{quitvmode} when capturing in a horizontal box.
3966 \end{enumerate}
3967
3968 You might wonder why the construction of the extern pages and the cc-memo (and
3969 actually, of the c-memo as well) is indirect, as described above.
3970 \begin{itemize}
3971 \item For one, the indirect construction facilitates potential abortion of
3972 memoization (see section~\ref{sec:limits}). With the indirect route,
3973 aborting is easy --- as nothing was permanently written anywhere yet, Memoize
3974 simply skips the final part of the process, where extern boxes are shipped
3975 into extern pages and the memo registers written into memo files --- and also
3976 clean: if \refcmd{mmzExternalizeBox} immediately shipped out the extern
3977 pages, these pages would remain in the document even in the case of abortion.
3978 \item Even more importantly, the cc-memo filename contains the \meta{context
3979 md5sum} (see section~\ref{sec:cc-memos}), but the context expression is not
3980 yet fully known when memoization starts --- remember (from
3981 section~\ref{sec:cross-referencing}) that a \cs{ref} in the memoized code will
3982 update the context! The cc-memo can therefore only be opened at the end of
3983 memoization, which necessitates a buffer (i.e.\ the \refcmd{mmzCCMemo}
3984 register) for storing its contents during memoization.
3985 \end{itemize}
3986
3987
3988 \subsubsection{Pure memoization}
3989 \label{sec:pure-memoization}
3990
3991 The default memoization driver discussed above is really an externalization
3992 driver: it produces a single extern. We now move to examples of drivers with
3993 other functions, starting with a pure memoization driver, which does not
3994 externalize any typeset output --- simply because it does not call
3995 \refcmd{mmzExternalizeBox} at any point --- but rather remembers the result
3996 of a (pgfmath) computation (let's pretend that the computation is
3997 time-consuming).
3998
3999 \begingroup
4000 \input{\exampledir pgfmathparse.ccmemo}
4001 \tcbinputexample{pgfmathparse}{%
4002 comment={%
4003 \tcbinputexample{pgfmathparse}{
4004 listing only, no attachment,
4005 width=.4\linewidth, nobeforeafter,
4006 listing file=\exampledir\mymemo,
4007 title=the cc-memo,
4008 }\quad
4009 \includeexamplepdf[document page]{page=1}
4010 },
4011 }
4012 \endgroup
4013
4014 Command \refcmd{mmz} above memoizes its mandatory argument with the memoization
4015 \refmmz{driver} set to the previously defined macro \cs{mmzPgfmathDriver}.
4016 Just as the default driver above, \cs{mmzPgfmathDriver} first executes the
4017 given code. However, there is no need to do this in the context of a
4018 |\setbox|, as the memoized code, which is obviously expected to consist of a
4019 single \cs{pgfmathparse} call, does not typeset anything: \cs{pgfmathparse}
4020 evaluates the given expression and stores the result into macro
4021 \cs{pgfmathresult}. The driver has two jobs: first, it must store this result
4022 into the cc-memo, to be utilized in subsequent compilations; second, because
4023 the assignment to \cs{pgfmathresult} (within \cs{pgfmathparse}) is local, the
4024 driver also needs to somehow smuggle the result out of the \cs{endgroup} issued
4025 by \refcmd{Memoize} and thereby make it into the following code (the final line
4026 of the example, which typesets the equation). Both jobs are easy enough: the
4027 expansion of |\def\noexpand\pgfmathresult{\pgfmathresult}| (in this case,
4028 |\def\pgfmathresult{42.0}|) is (globally) appended both to the token register
4029 \refcmd{mmzCCMemo}, which Memoize later writes into the cc-memo, and to the
4030 macro underlying the \refmmz{after memoization} hook, whose contents are
4031 executed \emph{after} closing the memoization group.
4032
4033 Let us consider an alternative implementation of the same goal of memoizing the
4034 result of a pgfmath computation, showcasing a couple of useful tricks.
4035
4036 \tcbinputexample{pgfmathparse-embellished}{listing only}
4037
4038 For one, this ``embellished'' example reminds us that we can list the
4039 \refmmz{driver} key among the auto-options (even if I don't really recommend
4040 automemoizing \cs{pgfmathparse}). But even more importantly, the example shows
4041 that the driver consist of more than a single control sequence; the only
4042 requirement is that the given driver code will consume the memoized code. In
4043 this example, we have developed a generic smuggling driver and applied it to
4044 \cs{pgfmathresult} in particular --- \cs{pgfmathresult} will become the first
4045 argument of \cs{mmzSmuggleOneDriver}, and the memoized code will become its
4046 second argument.
4047
4048 In the first version of the example, we have appended the same code to macro
4049 \refcmd{mmzCCMemo} and to macro \refcmd{mmzAfterMemoizationExtra} --- no
4050 surprise here, as we want the effect of memoization and utilization to be the
4051 same. In the embellished version, we advertise another way to achieve the same
4052 effect, a way which might be useful for complicated drivers: we simply smuggle
4053 out the entire cc-memo. The idea works even when memoization produces externs;
4054 in that case, however, the driver also has to say \cs{mmzkeepexternstrue} ---
4055 conditional \refcmd{ifmmzkeepexterns} decides whether Memoize keeps the externs
4056 around, in memory, even after shipping them out (but they are always gone at
4057 the start of the next memoization).
4058
4059 Finally, remember that the default context expression contains the
4060 \refmmz{padding} values. However, these really have no place in the context
4061 expression of some purely memoized code. We have therefore emptied out the
4062 context expression using \refmmz{clear context}.
4063
4064
4065 \subsubsection{Multiple externs per memo}
4066 \label{sec:memoization-multiple-externs}
4067
4068 In the next example, we show how to produce multiple externs for a single piece
4069 of memoized code. The usage case I find most appealing is breaking the typeset
4070 material, like a table, across pages --- but of course, table-breaking is too
4071 complicated an example, so we illustrate the idea by defining command
4072 |\countdown|, which counts down from the given number, typesetting each number
4073 into its own line. Clearly, if we were to externalize a call to this command
4074 using the default memoization driver, page breaking would stop working, as the
4075 entire countdown would be seen as a single, unbreakable box. To externalize it
4076 properly, the chunks of the countdown that should appear on separate pages must
4077 be externalized into separate externs, as shown below.
4078
4079 \tcbinputexample{countdown}(.c1){%
4080 listing options app={
4081 basicstyle=\ttfamily\footnotesize,
4082 },
4083 middle=1mm, bottom=0mm,
4084 comment={\centering
4085 \relaxmmzcommands
4086 \def\mmzUsedExtern##1{%
4087 \saveexpandmode\fullexpandarg
4088 \StrGobbleRight{##1}{4}[\mytemp]%
4089 \StrRight{\mytemp}{5}[\mytemp]%
4090 \restoreexpandmode
4091 \relaxmmzcommands
4092 }%
4093 \def\mmzNewExtern##1##2##3##4{\mmzUsedExtern{##1}}
4094 \def\myexternname##1{extern \dots\texttt{\mytemp##1.pdf}}%
4095 \input{\examplepath.mmz.c1}%
4096 \begin{tabular}{ccc}
4097 \includeexamplepdf[extern page,title/.expand once=\myexternname{}]
4098 {page=2,trim=1in 1in 1in 1in,scale=0.333}&
4099 \includeexamplepdf[extern page,title/.expand once=\myexternname{-1}]
4100 {page=3,trim=1in 1in 1in 1in,scale=0.333}&
4101 \includeexamplepdf[extern page,title/.expand once=\myexternname{-2}]
4102 {page=4,trim=1in 1in 1in 1in,scale=0.333}\\
4103 \includeexamplepdf[document page]{page=1,scale=0.333}&
4104 \includeexamplepdf[document page]{page=5,scale=0.333}&
4105 \includeexamplepdf[document page]{page=6,scale=0.333}
4106 \end{tabular}
4107 },
4108 }
4109
4110 To achieve this, we will have to integrate the memoization driver into the very
4111 code of |\countdown|. This approach contrasts sharply with the standard
4112 memoization driver, which is simply wrapped around the memoized code. Let's
4113 say we have implemented a non-memoization-aware variant of |\countdown| as a
4114 loop which gathers the countdown lines into a vertical box, and periodically,
4115 when this box holds all the material that will fit onto the page, places it
4116 into the main vertical list (i.e.\ on the page).\footnote{Of course,
4117 implementing \cs{countdown} this way would be idiotic; a sane implementation
4118 would simply spit out the countdown lines one by one, and let \hologo{TeX}
4119 deal with page-breaking. However, remember that we are pretending that we
4120 are typesetting (and page-breaking) some complex material, like a table; in
4121 such a case, the loop outlined in the main text would make perfect sense.}
4122 To have this command support memoization, we have to externalize our box every
4123 time we're placing it into the main vertical list. This is precisely what we
4124 do in the definition of |\countdowntypeset| below:\footnote{We omit the
4125 definition of the core algorithm of \cs{countdown} in the listing, because it
4126 is mostly irrelevant for our discussion, and only show the
4127 memoization-related code.} the final line of this macro adds the material to
4128 the main vertical list, and the preceding lines externalize it (the two lines
4129 inside \cs{ifmemoizingcountdown} should be familiar from the definition of the
4130 standard memoization driver; we'll explain about the conditional below); note
4131 that Memoize automatically deals with the fact that our box is vertical. As a
4132 result of having our memoization driver integrated into the loop of the core
4133 command, we can create as many externs as necessary, complete with the code in
4134 the cc-memo for including each and every one of them on subsequent
4135 compilations. Each extern eventually makes it into its own extern file, and
4136 note that the filenames of the non-first externs have their sequential number
4137 (we start counting at $0$) appended to the basename, as shown in the
4138 example.\cprotect\footnote{The \refmmz{auto} declaration of |\countdown| adds
4139 some relevant parameters to the context (see
4140 section~\ref{sec:cross-referencing}). The countdown will be recompiled upon
4141 change of either the font size (|f@size|), the text height (|\textheight|),
4142 or the height of the material in the main vertical list collected so far
4143 (|\pagetotal|). The |\pagetotal| parameter is especially important;
4144 including it makes sure that the countdown will be recompiled when it is
4145 pushed up or down the page. Also note that we want the context to record the
4146 value of |\pagetotal| when the (automemoized) |\countdown| is encountered
4147 (rather than at the end of memoization), so we expand it when applying the
4148 auto-options.}
4149
4150 \tcbinputexample{countdown}[.sty](.c1){listing only, after title pre={\ (version 1)}}
4151
4152 Of course, the chunks of the countdown should only be externalized when the
4153 code is actually being memoized, and not, say, when Memoize is disabled or
4154 performing regular compilation. (Note that this is a problem that only affects
4155 integrated drivers and not wrapped drivers such as the default driver.) The
4156 first thought is to detect whether we're undergoing memoization using
4157 conditional \refcmd{ifmemoizing}, which Memoize sets to true at the start of
4158 every memoization. This conditional is used in the run conditions of advice
4159 for \cs{ref} and \cs{label}, the idea being that they should add stuff to the
4160 context (\cs{ref}) and the cc-memo (\cs{label}) only when undergoing
4161 memoization. However, deploying \refcmd{ifmemoizing} in the current example
4162 would not be exactly right. It would work well with the main document as it
4163 is, but it would fail if |\countdown| was called from a piece of code that was
4164 independently submitted to memoization, e.g.\
4165 \refcmd{mmz}|{\countdown{30}}|.\footnote{Such embedding occurs more often than
4166 you might think. For example, \env{forest} calls \env{tikzpicture} under the
4167 hood, and both environments are automemoized.} In that case, both the
4168 \refcmd{mmz} driver and the |\countdown| integrated driver would get executed,
4169 resulting in the creation (and in subsequent compilations, utilization) of four
4170 externs: first, the |\countdown| driver would externalize each countdown chunk
4171 separately, and then, the \refcmd{mmz} driver would externalize them, all
4172 together, yet again. You can try this out by replacing
4173 \cs{ifmemoizingcountdown} in \cs{countdowntypeset} by \refcmd{ifmemoizing} (and
4174 wrapping the \cs{countdown} call in \cs{mmz}).
4175
4176 The solution to the \refcmd{ifmemoizing} problem deployed in the example is to
4177 declare a new, \cs{countdown}-specific memoization conditional, and set it to
4178 true in \cs{countdown}'s formal driver, i.e.\ the macro set as the
4179 \refmmz{driver} in the \refmmz{auto} declaration for \cs{countdown}. In fact,
4180 Memoize can do most of this for you: when we write \refmmzauto{integrated
4181 driver}|=countdown|, Memoize creates the countdown-specific memoization
4182 conditional and declares the formal driver which sets this conditional to true;
4183 you only have to access this conditional in your code, and you should do this
4184 using the \hologo{LaTeX}-style conditional \refcmd{IfMemoizing}, as shown
4185 below.\footnote{You shouldn't directly use the plain \hologo{TeX}
4186 countdown-specific conditional created by \refmmzauto{integrated driver} ---
4187 to prevent accidental access, Memoize doesn't actually name it
4188 \cs{ifmemoizingcountdown} --- because this conditional is undefined when
4189 Memoize is not loaded, i.e.\ when only package \refpkg{memoizable} is in
4190 effect. Furthermore, \refcmd{IfMemoizing} addresses a problem faced by
4191 integrated drivers of potentially recursive commands; we will talk about this
4192 in section~\ref{sec:memoization-complex-single-driver}.}
4193
4194 \tcbinputexample{countdown}[.sty](.c2){
4195 listing only,
4196 after title pre={\ (version 2)},
4197 attachment=countdown-integrated-driver.sty,
4198 }
4199
4200
4201 \subsubsection{Driver-based memoizable design}
4202 \label{sec:memoization-complex-single-driver}
4203
4204 In the previous section, we used the integrated driver approach to produce
4205 memos including multiple externs, but the approach can be useful for one-extern
4206 memos as well, when the extern must be integrated into the document in some
4207 special way. We already discussed such situations in
4208 section~\ref{sec:memoizable-design}, where we suggested to split a
4209 ``difficult'' command into the outer command and the inner command, and only
4210 submit the inner command to automemoization. However, the vanilla flavour of
4211 this approach had a negative impact on the user interface to automemoization.
4212 In this section, we will deploy the memoization driver to overcome the issue.
4213
4214
4215 Let us revisit the |poormansbox| example from
4216 section~\ref{sec:memoizable-design}. Remember that that environment produced
4217 a potentially framed box of a certain width, surrounded by some pre- and
4218 post-code, and that the issue was that the pre- and the post-code should not be
4219 memoized, but rather executed at every invocation of the command, as it was
4220 primarily intended to put some stretchable vertical space around the box.
4221
4222 The document
4223 source\attachexample[poormansbox-driver.tex][\exampledir poormansbox.tex.attachment]
4224 and the resulting PDF of the example are the same as in
4225 section~\ref{sec:memoizable-design}, so we will not repeat them here, but jump
4226 directly into a revised definition of the environment. We will retain the core
4227 idea from the original implementation: the outer command will execute the pre-
4228 and the post-code, and the inner command will typeset the box. But unlike in
4229 the original implementation, we will not automemoize the inner, internal
4230 command (this was the source of the author's discomfort) but the outer,
4231 user-level command --- and we will equip it with a custom memoization driver.
4232 The major idea here is to have the driver compose a cc-memo which not only
4233 includes the extern, but also executes the outer command.
4234
4235 \tcbinputexample{poormansbox-driver}[.sty]{listing only, float}
4236
4237 In detail, the implementation (partially shown in the |.sty| listing) is as
4238 follows. The outer command (|\poormansbox@outer|) first applies the options
4239 (|#1|) and then wraps the pre-code (|\pmb@before|) and the post-code
4240 (|\pmb@after|) around some arbitrary code (|#2|). During memoization or
4241 regular compilation, the outer command is invoked through the |poormansbox|
4242 environment, and you can see that in the definition of that environment, the
4243 second argument to |\poormansbox@outer| is a call to the inner command
4244 (|\poormansbox@inner|; this command takes two arguments, the options and the
4245 environment body). During utilization, the outer command is invoked from the
4246 cc-memo,\cprotect\footnote{As you can see, in the cc-memo the outer command is invoked
4247 by |\csuse{poormansbox@outer}|. A straight |\poormansbox@outer| would not
4248 work because we're in the middle of the document where |@| is not a letter,
4249 and including a |\makeatletter| in front of it (and in a group) only works if
4250 \refmmz{direct ccmemo input} was in effect. Under the default, indirect
4251 cc-memo input regime, the core cc-memo is tokenized before |\makeatletter| can
4252 take effect.} and as you can see in the cc-memo listing below, the second
4253 argument to |\poormansbox@outer| there is a call to \refcmd{mmzIncludeExtern}.
4254
4255 \begingroup
4256 \input{\exampledir poormansbox-driver.ccmemo}
4257 \tcbinputexample{poormansbox-driver}[.ccmemo]{
4258 listing only, left=1.5mm, right=1mm,
4259 listing options app={basicstyle=\tt\small},
4260 title=the second poor man's box's cc-memo,
4261 no attachment,
4262 }%
4263 \endgroup
4264
4265 And how does |\poormansbox@outer| get into the cc-memo, which normally only
4266 includes a call to \refcmd{mmzIncludeExtern}, you ask? This is the job of the
4267 memoization driver, which is in this case integrated into the inner command.
4268 The overall shape of the driver is the same as the shape of the standard
4269 driver, discussed in section~\ref{sec:memoization-driver-default}: typeset the
4270 extern material into a box, externalize this box, append the extern-inclusion
4271 code to the cc-memo, and put the extern box into the document. It is the
4272 cc-memo part which interests us right now: unlike the standard driver, we don't
4273 simply append the contents of |\mmz@temptoks|, i.e.\ a
4274 \refcmd{mmzIncludeExtern} call; we rather append a call to
4275 |\poormansbox@outer|, which gets the \refcmd{mmzIncludeExtern} call as its
4276 second argument (and the options as its first argument).
4277
4278 The core part of the driver, which externalizes the box and appends to the
4279 cc-memo, is embedded inside the true branch of conditional
4280 \refcmd{IfMemoizing}|[1]{pmb}|. We already used this conditional in
4281 section~\ref{sec:memoization-multiple-externs}, but without the optional
4282 argument. Such usage will not work here, because it is not recursion-safe.
4283 Unlike in the \cs{countdown} situation, one |poormansbox| environment can be
4284 embedded in another one (and, in our example, it is). If we deployed
4285 \refcmd{IfMemoizing}|{pmb}| in the inner command, the driver would be executed for
4286 \emph{both} the outer and the inner instance of the environment, whereas it
4287 should really be executed only for the outer instance.
4288
4289 When used in a recursion-safe way, i.e.\ with the optional argument,
4290 \refcmd{IfMemoizing} first tests whether the auxiliary command-specific
4291 conditional from the previous section is true, and then proceeds to compare the
4292 current group level (\hologo{eTeX}'s \refcmd{currentgrouplevel}) to the group
4293 level at the start of memoization (which Memoize stored in
4294 \refcmd{memoizinggrouplevel}). Only if these group levels match do we know
4295 that we're working on the outer instance of the environment, and that we should
4296 therefore execute the memoization driver. Importantly, though, the two group
4297 levels are compared modulo the offset, given as the optional parameter to
4298 \refcmd{IfMemoizing}: in our example, the offset is 1, because the driver is
4299 located inside the |poormansbox| environment, which opens a group --- note that
4300 0 zero (no offset) is not the default optional parameter; the absence of the
4301 optional parameter indicates that the non-recursion safe method should be
4302 used.\footnote{Even this approach is not completely bullet-proof. It will only
4303 work when the inner instance of the command is guaranteed to occur in an
4304 additional group, i.e.\ when our command opens up a group for any free-form
4305 code. I will assume that situations which require externalization of a
4306 potentially recursive command which, for some reason, cannot open the group
4307 before processing a free-form argument, are rare enough to not warrant a
4308 generic solution here.}
4309
4310
4311 \FloatBarrier
4312
4313
4314 \subsubsection{Shipout}
4315
4316 Memoize is a hypocrite: when it is creating extern pages, it uses
4317 \refcmd{primitive}\refcmd{shipout} to bypass the regular shipout routine of the
4318 format, but it is offended if anyone else does that.
4319
4320 Memoize bypasses the regular shipout because the extern pages should really not
4321 be modified or discarded by a foreign package. But using the primitive
4322 \refcmd{shipout} means that extern shipouts can't be detected by another
4323 package, at all. To facilitate peaceful coexistence with a potential package
4324 which needs to know about our extern pages, we offer public counter
4325 \refcmd{mmzExternPages} holding the number of externs shipped out so far. And
4326 if anyone really needs to \emph{do} something at every extern shipout, they can
4327 always (ab)use \refmmz[show keypath]{record/record type/new extern} as a
4328 post-extern-shipout hook.
4329
4330 The other side of the story is about Memoize needing to know the ``physical''
4331 page numbers of its externs in the document PDF --- how else are we to extract
4332 them? Memoize computes these page numbers by adding the values of several
4333 counters: \refcmd{mmzRegularPages}, which holds the number of regular shipouts;
4334 the above-mentioned \refcmd{mmzExternPages}, which holds the number of extern
4335 shipouts; and \refcmd{mmzExtraPages}, which holds the number of other shipouts.
4336 The latter counter should be advanced by a package which, like Memoize,
4337 bypasses the regular shipout routine.
4338
4339 \hologo{LaTeX} and \hologo{ConTeXt} kindly provide the number of regular
4340 shipouts as publicly a accessible counter, so we define
4341 \refcmd{mmzRegularPages} as synonymous with their
4342 \refcmd{ReadonlyShipoutCounter} and \refcmd{realpageno}. In \hologo{plainTeX}, we
4343 have to hijack the \refcmd{shipout} control sequence and count regular shipouts
4344 ourselves; as we have to hijack it while it still refers to the
4345 \refcmd{shipout} primitive, this format provides another reason for preferring
4346 Memoize to be loaded early.
4347
4348
4349 \subsection{Automemoization}
4350 \label{sec:tut:automemoization-details}
4351
4352 Automemoization is a mechanism that automatically memoizes the result of the
4353 compilation of certain commands and environments. Writing Memoize, I went to
4354 great lengths to make it flexible, yet easy to use. This resulted in
4355 automemoization deploying two specifically developed auxiliary packages:
4356 package Advice, which provides a generic framework for extending the
4357 functionality of selected commands and environments, and package CollArgs,
4358 which provides a command for collection of the arguments conforming to the
4359 given (slightly extended) \hologo{LaTeX3}\slash \pkg{xparse} argument
4360 specification.
4361
4362 This section lists the considerations which went into designing the system,
4363 followed by short tutorials on both auxiliary packages, which include several
4364 examples of how Memoize uses the underlying advising framework.
4365
4366 \paragraph{(De)activation}
4367 Ideally, all commands and environments where memoization makes sense would
4368 support Memoize (or Memoize would support them) and nothing would ever go
4369 wrong. In this dream world, memoization would be completely transparent to the
4370 author. However, things will go wrong, so at the very least, we need to
4371 offer the author a \emph{simple} way to \emph{selectively} switch automemoization
4372 on and off. This is achieved by keys \refmmz{activate} and
4373 \refmmz{deactivate}.
4374
4375 \paragraph{Submission}
4376 Of course, there will be commands without official support by either Memoize or
4377 the package which defines them; clearly, at the moment when I write this, all
4378 commands but \cs{tikz}, \env{tikzpicture} and \env{forest} are such. Or, the
4379 author might want to automemoize his or her own command. Ideally, submitting a
4380 new command to automemoization would be as simple as
4381 |memoize=|{\meta{command}}, and for environments, this is in fact achievable,
4382 although the actual interface is
4383 \refmmz{auto}|=|\braces{\meta{environment}}\braces{\refmmzauto{memoize}}. But
4384 simply submitting the name cannot work for commands, because commands are where
4385 we encounter the major \hologo{TeX}nical problem with automemoization: we need
4386 to somehow collect the arguments of the command --- without executing the
4387 command itself.
4388
4389 \paragraph{Argument collection using CollArgs}
4390 \hologo{TeX} being \hologo{TeX}, automatically determining the scope of a
4391 command in general is just plain impossible. Note that inspecting the
4392 |\meaning| is not enough in general, because the ``real'' and the formal
4393 arguments of a command can, and quite often do, differ wildly. The author (or the
4394 package writer) will need to tell Memoize about the argument structure of the
4395 command. And as there is already a nice and general argument specification on
4396 the market --- I'm obviously referring to the argument specification of package
4397 \pkg{xparse}, which was recently integrated into the core \hologo{LaTeX}
4398 --- why not use that? Memoize comes with an auxiliary package CollArgs, which
4399 (given the slightly extended \pkg{xparse}-style argument specification)
4400 collects the arguments of a command into a single entity. All the user needs
4401 to write to enable automemoization for a command is thus
4402 \refmmz{auto}|=|\meta{command}\braces{\refmmzauto{memoize},
4403 \refmmzauto{args}|=|\braces{\meta{argument specification}}}. Even simpler,
4404 when it comes to commands defined by \hologo{LaTeX3}'s \refcmd{NewDocumentCommand}
4405 or friends, writing \refmmz{auto}|=|\meta{command}\braces{\refmmzauto{memoize}}
4406 will suffice, as the argument specification of these commands can be retrieved
4407 by \refcmd{GetDocumentCommandArgSpec}.
4408
4409 \paragraph{Weird commands}
4410 Not every argument structure can be described using \pkg{xparse}'s argument
4411 specification, a case in point being \refcmd{tikz} with its totally
4412 idiosyncratic syntax --- and if Memoize won't support \cs{tikz}, why have it at
4413 all? The interface to automemoization must be flexible enough to cover even
4414 the craziest commands, and this is why Memoize allows for arbitrary argument
4415 collectors. These are defined by the advanced user or package writer and then
4416 declared to be used for parsing the argument structure of a command by writing
4417 \code{\refmmz{auto}=\marg{command}\bracestt{...,\refmmzauto{collector}=\meta{argument
4418 collector}}}.
4419
4420 \paragraph{Over and above automemoization: handlers}
4421 The framework facilitating automemoization must cover more than just that. For
4422 one, it sometimes makes sense to automatically \emph{prevent} memoization
4423 during the execution of certain commands (as in the \refmmzauto{nomemoize}
4424 example in section~\ref{sec:tut:redefinitions}). It follows that the action
4425 performed to an invocation of a command should not be fixed. In the advising
4426 framework, implemented by the auxiliary package Advice, we assign each
4427 advised command a handler --- a command which does the real work of
4428 memoizing or whatever. Crucially, the handler and the collector are
4429 independent of each other, allowing a single memoization handler to handle
4430 commands with both standard and non-standard argument structure, and allowing a
4431 single collector to serve either the memoization or the no-memoization handler.
4432
4433 \paragraph{Over and above automemoization: the outer and the inner handler}
4434 Second, unlike the memoization handler set by \refmmzauto{memoize}, not all
4435 handlers work with the entire argument list of the advised command. Some
4436 handlers don't care about the arguments of the advised command at all:
4437 \refmmzauto{abort} simply aborts memoization whenever the advised command is
4438 executed. Other handlers are only intended to advise a single command or a
4439 small family of commands, and need to inspect specific arguments of the advised
4440 command: for example, \refmmzauto{ref} needs to append the internal
4441 cross-reference macro to the context, and it of course constructs the name of
4442 this macro from the reference key. For all such handlers, it would be plain
4443 wasteful to first collect the arguments and then tear them apart to inspect
4444 them (or not). The advising framework therefore recognizes two kinds of
4445 handlers. The abortion and the cross-reference handler are examples of an
4446 \emph{outer} handler, which is simply placed in front of the arguments of the
4447 handled command as they are, without invoking the collector. The memoization
4448 handler, on the other hand, is an example of an \emph{inner} handler, which
4449 receives the entire argument list from the collector (as a single argument) ---
4450 or more precisely, even \refmmzauto{memoize} sets up an outer handler, but this
4451 outer handler doesn't do much more than invoke the collector, which in turn
4452 invokes the inner handler.
4453
4454 \paragraph*{Run conditions} are another, if minor, lego piece of the
4455 advising framework. Using key \refmmzauto{run conditions}, the user can set the
4456 conditions under which the (outer) handler is executed; for example,
4457 cross-reference commands are only advised when memoization is underway. And
4458 the same goes for \refcmd{label}, and for the \refmmzauto{replicate}d
4459 \refcmd{index}, and for \refcmd{savepos}, upon which memoization must be
4460 aborted. The bottom-line is that run conditions repeat across handlers, so
4461 it makes sense to separate them out as an independent component of the
4462 framework, with the added bonus that the system can make sure that an
4463 invocation of a advised command which does not satisfy the run conditions
4464 will incur as little overhead as possible.
4465
4466 \paragraph{Bailout handler}
4467 An automemoized command applies the next-options (set by \refcmd{mmznext}), but
4468 what happens when the run conditions are not satisfied? If nothing happened,
4469 the existing next-options might apply to the next instance of
4470 (auto)memoization, which would not be what the author intended. This is why
4471 Advice introduces the bailout handler, a piece of code executed before the
4472 original command when the run conditions are not met. Obviously, the bailout
4473 handler for memoization clears out the next-options (and does not process
4474 them).
4475
4476 \paragraph{The structure of advice}
4477 Together, the components mentioned above form a piece of \emph{advice}:
4478
4479 \begin{center}
4480 \begin{forest}
4481 /tikz/edge node/.style={midway, font=\scriptsize, yshift=0.5ex},
4482 for tree={l sep=5ex, % manual, to avoid a page break
4483 child anchor=north, s sep+=2em},
4484 [activated?
4485 [\refmmzauto{run conditions}?,
4486 edge=dashed, edge label={node[edge node, left]{yes}}
4487 [\refmmzauto{outer handler}\meta{args},
4488 edge=->, edge label={node[edge node, left]{yes}},
4489 [default:,
4490 edge=dashed,
4491 [\refmmzauto{collector}\meta{args}, edge=->,
4492 [\refmmzauto{inner handler}
4493 \textcolor{red}{\bracestt{\textcolor{black}{\meta{args}}}},
4494 edge=->
4495 ]
4496 ]
4497 ]
4498 [other:\\whatever\rlap\footnotemark,
4499 align=center, edge=dashed]
4500 ]
4501 [\refmmzauto{bailout handler},
4502 edge=->, edge label={node[edge node, right]{no}},
4503 [\meta{original command}\meta{args}, edge=->]
4504 ]
4505 ]
4506 [\meta{original command}\meta{args},
4507 edge=dashed, edge label={node[edge node, right]{no}},
4508 ]
4509 ]
4510 \end{forest}%
4511 \footnotetext{The handler may do whatever as long as it consumes all and
4512 only the arguments of the original command.}
4513 \end{center}
4514
4515
4516
4517 \paragraph{Deferred activation}
4518 Memoize needs to be loaded early, but activation should take place late, so
4519 that it can surely override the submitted commands; a case in point,
4520 \pkg{hyperref} redefines \refcmd{ref} very late. To address the issue, the
4521 advising framework implements the deferred \refmmz{activation} regime, under
4522 which (de)activation commands are not executed but collected in a special
4523 style, \refmmz{activate deferred}. Memoize deploys the deferred activation
4524 regime throughout the preamble, and executes \refmmz{activate deferred} at the
4525 latest possible \docref{hook:begindocument} hook; as a bonus, it also offers
4526 the author a way to avoid automatic activation completely by invoking key
4527 \refmmz{manual}.
4528
4529
4530 \paragraph{Install anywhere}
4531 Once all this machinery is developed, why not offer it to others as well?
4532
4533 Once I decided to offer Advice (at the time, still called Auto) as a standalone
4534 package (and I freely admit that the framework got much cleaner once I
4535 separated its code out of Memoize) it became immediately clear that if it is
4536 to serve as a generic framework, it should be possible for multiple packages to
4537 use it without interfering with each other. The package thus allows any number
4538 of installations into different namespaces, each namespace a \pkg{pgfkeys}
4539 keypath. The installation is a breeze:
4540 \cs{pgfkeys}\bracestt{\meta{namespace}/\refkey{/handlers/.install advice}}.
4541
4542
4543 \subsubsection{Using package Advice}
4544 \label{sec:advice}
4545
4546 In this section, we will provide some examples of handler declarations, mainly
4547 based on how Memoize deploys the advising framework.
4548
4549 \paragraph{\refmmzauto[show keypath]{memoize}}
4550
4551 In section~\ref{sec:tut:automemoization}, the author was instructed to submit a
4552 command to automemoization by writing
4553 \code{\refmmz{auto}=\meta{command}\bracestt{\refmmzauto{memoize},...}}. The
4554 auto-key \refmmzauto{memoize} is a style (defined by Memoize rather than
4555 Advice) which sets the appropriate components of the automemoization advice.
4556 Residing in keypath \refkeypath{/mmz/auto}, it is effectively defined as
4557 follows:
4558
4559 \begin{tcblisting}{listing only}
4560 \mmzset{
4561 auto/memoize/.style={
4562 run if memoization is possible,
4563 bailout handler=\mmz@auto@bailout,
4564 outer handler=\mmz@auto@outer,
4565 inner handler=\mmz@auto@memoize
4566 }
4567 }
4568 \end{tcblisting}
4569
4570 The heart of this advice is its inner handler, which actually triggers
4571 memoization by executing \refcmd{Memoize}. Remember that the first argument of
4572 \refcmd{Memoize} is the code which the md5sum is computed off of. This
4573 argument must therefore be identical to the author's invocation of the
4574 automemoized command or environment. Given what Advice offers, this is easy to
4575 construct: \refcmd{AdviceReplaced} holds the code replaced by the advice, and
4576 the \meta{arguments} of the automemoized command are waiting for us in |#1|.
4577 The second argument of \refcmd{Memoize} will be similar, but as this is the
4578 code which will get compiled, we have to execute the original definition of the
4579 command, followed by the (unbraced!) \meta{arguments} as |#1|; this is a job
4580 for \refcmd{AdviceOriginal}. (Note that \emph{executing}
4581 \refcmd{AdviceReplaced} would run the auto-handler again, resulting in an
4582 infinite loop! Or at least a pile of errors.)
4583
4584 However, the overly simplistic approach shown below won't necessarily work.
4585 The issue is that the arguments of \refcmd{Memoize} contain
4586 \refcmd{AdviceReplaced} and \refcmd{AdviceOriginal} themselves, instead of their
4587 contents, i.e.\ (first) expansions.
4588
4589 \begin{tcblisting}{listing only, bad}
4590 \long\def\mmz@auto@memoize#1{%
4591 \Memoize{\AdviceReplaced#1}{\AdviceOriginal#1}%
4592 }
4593 \end{tcblisting}
4594
4595 Regarding the first argument, the problem is that the code md5sum will be
4596 computed off of the token list \refcmd{AdviceReplaced}\meta{arguments} ---
4597 exactly as you see it.\footnote{If you inspected a c-memo, you would find
4598 \refcmd{AdviceReplaced}\meta{arguments} in the \refcmd{mmzSource} section.}
4599 This implies that two commands sharing exactly the same \meta{arguments} will
4600 receive the same (c-)memo. For example, if you automemoized first
4601 |\textit{foo}| and then |\textbf{foo}|, both would come out as a bold ``foo''
4602 upon utilization.
4603
4604 The second argument illustrates a general issue about the lifespan of
4605 \refcmd{AdviceOriginal} and other |\Advice...| commands.\footnote{The list of
4606 all commands available only within the handler can be found in the
4607 documentation of key \refmmzauto{outer handler} in
4608 section~\ref{sec:ref:advice}.} By executing \refcmd{Memoize}, we leave the
4609 advice and thereby cannot be sure that when expanded, \refcmd{AdviceOriginal}
4610 will mean what it means at the moment. In general, another piece of advice might be
4611 triggered until its expansion, or the group might be closed, etc. For example,
4612 the author may have issued \refcmd{mmznext}\bracestt{\refmmz{at begin
4613 memoization}=\cs{label}\marg{key}}, and as the pre-memoization code is
4614 executed before the memoization driver, the \cs{label}, which is submitted to
4615 Advice so that any \cs{label}s inside the memoized code ``just work'', would
4616 execute another piece of advice, redefining \refcmd{AdviceOriginal} and friends.
4617 Effectively, you'd end up memoizing an invocation of the \cs{label} command.
4618
4619 The bottom line is that while the code following the above template
4620 \emph{might} sometimes work, Advice offers no guarantees that it will, so I
4621 advise against using it. The actual definition of the memoization inner
4622 handler is shown below. In this definition, we expand \refcmd{AdviceReplaced}
4623 and \refcmd{AdviceOriginal} --- exactly once! --- into the respective arguments
4624 of \refcmd{Memoize}; of course, as the entire invocation of \refcmd{Memoize} is
4625 expanded, we have to guard against expanding the collected arguments (|#1|) and
4626 \refcmd{Memoize} itself.\footnote{As the final touch, the handler also contains
4627 \refcmd{ignorespaces} after the invocation of \refcmd{Memoize}, if this was
4628 requested using \refmmz{ignore spaces}. Note that this could not be done
4629 without pre-expanding the \cs{ifmmz@ignorespaces} conditional, as
4630 \refcmd{Memoize} closes the group in which the auto- and the next-options are
4631 applied.} The result of the expansion is shown under the code: the first and
4632 the second line assume we are automemoizing command |\foo| and environment
4633 |bar|, respectively. Note that \refcmd{AdviceOriginal} expands into an
4634 invocation of \refcmd{AdviceGetOriginal}, a command which may be safely used
4635 outside the advice; the first argument of this command is the auto-namespace
4636 (in our case, \refkeypath{/mmz}), and the second argument is the advised
4637 command. For a \hologo{LaTeX} environment, the advised command is actually
4638 \refcmd{begin}, and this is why the call of \refcmd{AdviceGetOriginal} is of
4639 course followed by the environment name.
4640
4641 \tcbinputexample{_auto-memoize-inner}[.excerpt]{%
4642 listing and comment, no .listing, no attachment,
4643 title={The implementation of the \refmmzauto{inner handler}
4644 for automemoization},
4645 comment={%
4646 $\rightarrow$ \refcmd{Memoize}\braces{\cs{foo}\Arg1}%
4647 \braces{\refcmd{AdviceGetOriginal}\bracestt{/mmz}\bracestt{\cs{foo}}\Arg1}\\
4648 $\rightarrow$ \refcmd{Memoize}\braces{\cs{begin}\bracestt{bar}\Arg1}%
4649 \braces{\refcmd{AdviceGetOriginal}%
4650 \bracestt{/mmz}\bracestt{\cs{begin}}\bracestt{bar}\Arg1}
4651 }
4652 }
4653
4654 Let us now move backwards in time and look at the outer handler installed by
4655 \refmmzauto{memoize}. It is very simple, but performs an important function of
4656 applying the auto- and the next-options (in this order), which also
4657 necessitates opening a group (closed by \refcmd{Memoize}). The final line
4658 invokes the argument collector, which then calls the inner handler; remember
4659 that the invocation of \refcmd{AdviceCollector} is the sole function of the
4660 default outer handler.
4661
4662 \tcbinputexample{_auto-memoize-outer}[.excerpt]{%
4663 listing only, no .listing, no attachment,
4664 title={The implementation of the \refmmzauto{outer handler}
4665 for automemoization},
4666 }
4667
4668 Moving even further back in time, we arrive at the run conditions. The
4669 \refmmzauto{memoize} style invokes \refmmzauto{run if memoization is possible},
4670 defined as \code{run conditions=\cs{mmz@auto@rc@if@memoization@possible}}, with
4671 the installed macro as shown below. Indeed, memoization only makes when
4672 Memoize is \refmmz{enable}d (which we test using \refcmd{ifmemoize}), but we're
4673 not already ``inside \refcmd{Memoize}'' (which we test using
4674 \refcmd{ifinmemoize}). The latter condition is true when we're either
4675 memoizing or regularly compiling some code submitted to memoization (see the
4676 diagram in section~\ref{sec:Memoize}). Note that it is not necessary to invoke
4677 \refcmd{AdviceRunfalse} in branches where the run conditions are not satisfied.
4678
4679 \tcbinputexample{_auto-run-if-memoization-is-possible}[.excerpt]{%
4680 listing only, no .listing, no attachment,
4681 title={The implementation of
4682 \refmmzauto{run if memoization is possible}},
4683 }
4684
4685 While it is clear that double memoization is a no-no, why should we avoid
4686 memoizing inside a regular compilation? Imagine that Memoize decides not to
4687 memoize a Forest tree, perhaps because \refmmz{readonly} is in effect. Under
4688 the hood, Forest creates many \env{tikzpicture}s. Should all of them be
4689 (auto)memoized now? Certainly not.
4690
4691 Finally, what happens when the run conditions are not met? Not much, but
4692 something important nevertheless: by consuming the next-options, the
4693 \refmmzauto{bailout handler} makes sure they will not erroneously apply to the
4694 next instance of (auto)memoization.
4695
4696 \tcbinputexample{_auto-memoize-bailout}[.excerpt]{%
4697 listing only, no .listing, no attachment,
4698 title={The implementation of the \refmmzauto{bailout handler}
4699 for automemoization},
4700 }
4701
4702 The only component of the automemoization advice not determined by style
4703 \refmmzauto{memoize} is the argument collector, which allows the user to submit
4704 a command with a weird argument structure to automemoization simply by setting
4705 key \refmmzauto{collector} in addition to executing \refmmzauto{memoize}. For
4706 example, Memoize submits \refcmd{tikz} to automemoization by loading
4707 \reffile{advice-tikz.code.tex}, which contains Advice's definition of the
4708 \cs{tikz} collector \refcmd{AdviceCollectTikZArguments}, and issuing the
4709 following declaration.
4710
4711 \begin{tcblisting}{
4712 example title={Declaring automemoization of command \cs{tikz}},
4713 listing only,
4714 }
4715 auto=\tikz{memoize, collector=\AdviceCollectTikZArguments},
4716 \end{tcblisting}
4717
4718 \paragraph{\refmmzauto[show keypath]{ref}}
4719
4720 The cross-reference advice presents an example of an outer handler radically
4721 different from the default outer handler. This outer handler does not invoke
4722 the collector at all. As shown below, it grabs the argument of \refcmd{ref} (or
4723 whichever cross-referencing command) on its own --- remember that the outer
4724 handler receives the arguments of the handled command ``as they are,'' i.e.\
4725 uncollected. It then asks \refcmd{mmzNoRef} to do the real job of getting the
4726 reference key into the context, and finally executes the original \cs{ref}.
4727
4728 \tcbinputexample{auto-ref}{%
4729 listing only, no attachment,
4730 title={\hypercolor{link}{white}A simplified\footnotemark{} definition
4731 of \refmmzauto{ref}},
4732 % This only works if there is a single footnote in the memoized code.
4733 /mmz/context/.expanded={footnote=\numexpr\thefootnote-1},
4734 }\footnotetext{The real outer handler allows for arbitrary optional arguments of
4735 the cross-referencing command, and shares code with \refmmzauto{force
4736 ref}.}
4737
4738 The run conditions of this style are agonizingly simple: \refmmzauto{run if
4739 memoizing} sets \refmmzauto{run conditions} to a macro defined as
4740 \refcmd{ifmemoizing}\hyp\refcmd{AdviceRuntrue}\hyp\cs{fi}.
4741
4742 \paragraph{\refmmzauto[show keypath]{abort}}
4743
4744 The advice for aborting memoization is very simple --- it merely executes
4745 \refcmd{mmzAbort} --- but also very sneaky. Here, the \refmmzauto{run
4746 conditions} do the real work of aborting memoization, while the ``real,''
4747 i.e.\ outer handler, never even gets executed; note the absence of
4748 \refcmd{AdviceRuntrue}, which implies \refcmd{AdviceRunfalse}, which triggers
4749 the execution of the original command after the run conditions are ``checked.''
4750
4751 \tcbinputexample{_auto-abort}[.excerpt]{%
4752 listing only, no .listing, no attachment,
4753 title={The definition of \refmmzauto{abort}},
4754 }
4755
4756 The point here is that executing \refcmd{mmzAbort} (itself a single-liner
4757 setting an internal conditional) is cheaper than testing for the real run
4758 conditions (\refmmzauto{run if memoizing}) and aborting only if they are
4759 satisfied. Of course, the trick only works because (i) the advice doesn't
4760 need to inspect any arguments of advised command, and because (ii) setting the
4761 internal abortion conditional outside memoization does no damage.
4762
4763 \paragraph{Advice in chains}
4764
4765 A command may be submitted to several instances of the advising framework,
4766 i.e.\ instances installed under different keypaths. In the example below, we
4767 submit |\foo| both to the instance of Advice installed in keypath |/one| and to
4768 the one installed in keypath |/two|. Under |/one|, the result of |\foo{...}|
4769 will be boxed (|\fboxWrap|); under |/two|, in will be parenthesized
4770 (|\parenWrap|). The order in which this happens depends on the order in which
4771 |\foo| was activated under different keypaths. If we first activate it under
4772 |/one| with the boxing effect and then under two with the parenthesizing
4773 effect, the box will appear within parenthesis; if we reverse the activation
4774 order, the parenthesis will appear inside the box.
4775
4776 \tcbinputexample{chained-advice}{comment=\input{\examplepath}}
4777
4778 First of all, looking at the code above, you have probably noticed the absence
4779 of key \refmmz{auto}. This is because by default, \refkey{/handlers/.install
4780 advice} defines the \emph{setup key} to be |advice| --- Memoize overrides
4781 this default by installing the framework with \refkey{/handlers/.install
4782 advice}|=|\bracestt{\refkey{/advice/install/setup key}=auto, ...}.
4783
4784 Next, |advice'| is a variant of |advice| which prevents automatic activation
4785 upon setup (and the same holds for |auto'| vs.\ |auto| in Memoize). We have
4786 used the bar variant above to make it clear that it is the order of activation,
4787 rather than declaration by |advice|\slash \refmmz{auto}, which matters in
4788 determining which handler is applied first.
4789
4790 Finally, note that the deactivation order must be the reverse of the activation
4791 order. So if we activate |\foo| first in |/one| and then in |/two|, we should
4792 deactivate it in |/two| first and in |/one| next, otherwise Advice will
4793 complain.
4794
4795
4796 \paragraph{A simple collector}
4797
4798 Let us implement a collector for a command which accepts one (standard
4799 \hologo{LaTeX}) optional argument and one mandatory argument; in
4800 \hologo{LaTeX3} terms, a command with argument specification |om|.
4801
4802 Using \cs{NewDocumentCommand}, such a collector is very easy to implement. We
4803 simply define a command with signature |om| and distinguish two possibilities
4804 regarding the presence of the optional argument, which we test using
4805 \cs{IfValueTF}. If the true branch, we pass a \emph{braced} |[#1]{#2}| to the
4806 inner handler, which we invoke by \refcmd{AdviceInnerHandler}; in the false
4807 branch, we omit the optional argument, passing it an (additionally) braced
4808 |{#2}|.
4809
4810 \tcbinputexample{om-collector-NewDocumentCommand}{listing only}
4811
4812 Defining a functionally equivalent collector using \cs{newcommand} would be a
4813 bit more involved, as \hologo{LaTeX2e} does not offer a standardized way to
4814 test for the \emph{presence} of the optional argument. Consider the following
4815 collector, whose optional argument has the same default value as the advised
4816 command. Is it functionally equivalent to the one above?
4817
4818 \tcbinputexample{om-collector-newcommand}{listing only}
4819
4820 While there will be no visual difference, there is a difference under the hood.
4821 If you compile both documents, you will see that the first one creates three
4822 memos\slash externs, while the second one only creates two: |\foo{...}| does
4823 not have its own memo anymore, but creates and uses the same memo as
4824 |\foo[green]{...}|.
4825
4826 While the second version might sometimes be preferred, perhaps even in the
4827 context of memoization, the initial collector, which deploys command
4828 \refcmd{CollectArguments}, behaves like the \cs{NewDocumentCommand}-defined
4829 collector above,\footnote{In fact, there is a slight difference after all.
4830 While the above-defined collector won't distinguish between the single-token
4831 mandatory argument given with or without braces, \refcmd{CollectArguments}
4832 will again faithfully replicate the original argument tokens.} as it attempts
4833 to perfectly replicate the command invocation. Furthermore, this behaviour
4834 makes it unnecessary for the author to provide the default values of optional
4835 arguments (and even allows them to replace |O{default}| in the argument
4836 specification by \docref{xparse:o}).
4837
4838 We now turn to the package CollArgs, which implements the actual argument
4839 collection; we'll revisit the initial collector of Advice at the end of the
4840 following subsection.
4841
4842
4843
4844 \subsubsection{Using package CollArgs}
4845 \label{sec:collargs}
4846
4847 Automemoization is implemented on top of the framework offered by package
4848 Advice, and that package in turn couldn't really work as intended without
4849 package CollArgs. A regular user of Memoize shouldn't need to know anything
4850 about CollArgs, but a package writer wanting to support Memoize might have to.
4851
4852 The package provides two public commands, \refcmd{CollectArguments} and
4853 \refcmd{CollectArgumentsRaw}; we'll focus on the former first.
4854 \refcmd{CollectArguments} takes three arguments: optional \meta{options} in the
4855 form of a \pkg{pgfkeys} keylist; a mandatory \meta{argument specification} in a
4856 (slightly extended) \hologo{LaTeX3}\slash\pkg{xparse} format; and the
4857 \meta{next-code}:
4858 \begin{center}
4859 \refcmd{CollectArguments}\oarg{options}\marg{argument specification}\marg{next-code}%
4860 \textcolor{gray}{\meta{tokens}}
4861 \end{center}
4862 Following the three formal arguments of \refcmd{CollectArguments} are some
4863 \meta{tokens} --- the rest of the document, really --- and the job of
4864 \refcmd{CollectArguments} is to figure out the extent to which these
4865 \meta{tokens} conform to the given \meta{argument specification}. In other
4866 words, \refcmd{CollectArguments} will consume as many of the \meta{tokens} as a
4867 \meta{command} defined by \refcmd{NewDocumentCommand}\meta{command}\marg{argument
4868 specification}\bracestt{...} would. Once these \meta{argument tokens} are
4869 collected, \refcmd{CollectArguments} executes the \meta{next-code} with the
4870 \meta{argument tokens} given as a single, braced argument (clearly, the
4871 \meta{rest} of the \meta{tokens}, i.e.\ the non-consumed tokens, will follow):
4872 \begin{center}
4873 \meta{next-code}\marg{argument tokens}\textcolor{gray}{\meta{rest}}
4874 \end{center}
4875
4876 In the example below, we define macro |\PrintAndDo|, which takes two arguments,
4877 a command and the collected arguments of that command, prints out which command
4878 we're about to execute and with what arguments, and then executes that command
4879 with those arguments --- |#1#2| at the end of the definition. Note that |#2|
4880 immediately following |#1| is not braced, so
4881 |\PrintAndDo\makebox{[5em][r]{text}}| executes |\makebox[5em][r]{text}|.
4882
4883 {\tolerance 1000 Executing |\PrintAndDo\makebox{[5em][r]{text}}| directly would
4884 thus yield the first line of the result below --- and in fact, this is
4885 precisely what gets executed to yield that line, but in a roundabout fashion.
4886 Given the argument specification |oom| (two optional arguments followed by a
4887 mandatory argument), \refcmd{CollectArguments} figures out how many tokens
4888 following its formal arguments conform to this argument specification ---
4889 below, these would be |[5em][r]{text}| following
4890 \refcmd{CollectArguments}\hyp|{oom}{\PrintAndDo\makebox}|--- and puts them,
4891 braced, behind its \meta{next-code} argument, |\PrintAndDo\makebox|, yielding
4892 |\PrintAndDo|\hyp|\makebox{[5em][r]{text}}|.\par}
4893
4894 \tcbinputexample{collargs-makebox}{listing and compile}
4895
4896 Seeing the arguments of |\makebox| without the immediately preceding |\makebox|
4897 might seem strange, but remember that \refcmd{CollectArguments} is about the
4898 arguments of a command, not about the command's control sequence. It doesn't
4899 know or care which command the argument tokens ``belong'' to, as long as they
4900 conform to the given specification. In the example above, it is only in |#1#2|
4901 of |\PrintAndDo| that |\makebox| is ``reunited'' with its arguments, but note
4902 that the reunion is far from obligatory.
4903
4904 CollArgs supports all the argument types (and modifiers) that \hologo{LaTeX3}
4905 does, complete with the uncommon\slash historical types only available upon
4906 loading package \pkg{xparse}. Below, we provide an example of the environment
4907 type \docref{xparse:b}. Again, the code might seem strange, as it features an
4908 |\end{minipage}| without the matching |\begin{minipage}|, but the logic is
4909 similar as for commands: just as \refcmd{CollectArguments} occurs in front of
4910 the command arguments, without the command itself, so it occurs in front of the
4911 environment body, without the opening of that body. However, while
4912 \refcmd{CollectArguments} never needs to know the command name, we need to
4913 inform it of the environment name, so that it can find the end of the
4914 environment. This can be achieved as shown below, using key
4915 \refcollargs{environment} in the optional argument of the command, or by our
4916 extension to the \hologo{LaTeX3}\slash\pkg{xparse} argument specification,
4917 where the environment argument type \docref{xparse:b} may be followed by a
4918 braced environment name. In the example below, we could therefore also invoke
4919 argument collection by \refcmd{CollectArguments}\braces
4920 {\docref{xparse:+}\docref{xparse:b}\braces{\env{minipage}}} (we have preceded
4921 \docref{xparse:b} with a \docref{xparse:+} to allow for an environment body
4922 containing paragraph tokens).\footnote{CollArgs automatically adapts to the
4923 format, i.e.\ it knows that environments are tagged by \cs{}\meta{name} and
4924 \cs{end}\meta{name} in \hologo{plainTeX} and by \cs{start}\meta{name} and
4925 \cs{stop}\meta{name} in \hologo{ConTeXt}.}
4926
4927 \tcbinputexample{collargs-minipage}{comment=\input{\examplepath.tex}}
4928
4929 You might wonder why didn't we provide \refcmd{CollectArguments} in the
4930 previous example with argument specification |omb| --- after all, the
4931 \env{minipage} environment takes an optional and a mandatory argument. While that
4932 would work, and produce the same result,\footnote{Unless argument processing
4933 was in effect; see section~\ref{sec:ref:collargs} for details.} note that
4934 \refcmd{CollectArguments} is only interested in finding the scope of the
4935 arguments, and grabbing everything until |\end{minipage}| is the same as first
4936 grabbing the optional argument, maybe, then the mandatory argument, and finally
4937 the argument body.
4938
4939 \refcmd{CollectArguments} not only supports the verbatim argument
4940 type \docref{xparse:v}, it can grab an argument of \emph{any} type in the
4941 verbatim mode, triggered by option \refcollargs{verbatim}.\footnote{We refer to
4942 the verbatim mode triggered by \refcollargs{verbatim} as the full verbatim
4943 mode, where all characters are of category ``other''. There is also the
4944 partial verbatim mode, triggered by \refcollargs{verb}, where braces retain
4945 their normal category codes.} We illustrate this key below, where we also use
4946 option \refcollargs{tags}, which makes CollArgs automatically surround the
4947 grabbed environment body with the begin tag \refcmd{begin}\marg{environment
4948 name} and the end tag \cs{end}\marg{environment name}, and use
4949 \refcmd{scantokens} to execute the grabbed environment. Consult
4950 section~\ref{sec:ref:collargs} for the full reference on the verbatim mode and
4951 its limitations.
4952
4953 % .dtx has separate versions for listing and compilation, because compilation
4954 % of the multi-line version yields:
4955 % Missing character: There is no ^M (U+000D) in font [lmmono10-regular]:!
4956 % because \texttt in the example is fed a (verbatim) carriage return
4957 \tcbinputexample{collargs-verbatim}{listing and compile}
4958
4959 Finally, CollArgs extends the \hologo{LaTeX3}\slash\pkg{xparse} specification
4960 by modifier \docref{xparse:amp}, which allows the user to specify options which
4961 apply only to the following argument, as opposed to the options given as the
4962 optional argument of \refcmd{CollectArguments}, which apply to all the
4963 arguments. A third way to invoke the environment body collection in the above
4964 example is thus \refcmd{CollectArguments}%
4965 \bracestt{\docref{xparse:amp}\bracestt{\refcollargs{environment}=minipage}+b}.
4966
4967 Both the single-argument and the common options can be given not only as
4968 \pkg{pgfkeys} keys, but also in the raw, ``programmer's interface'' format.
4969 Every option key has a corresponding macro; for example, key
4970 \refcollargs{environment} is matched by macro \refcmd{collargsEnvironment}.
4971 The macros are listed alongside their corresponding keys in the reference
4972 section~\ref{sec:ref:collargs}; here, we merely learn how to use them.
4973
4974 To use raw options for a single argument, double the ampersand in the argument
4975 specification. Therefore, the fourth way to specify the environment name is
4976 \code{\docref{xparse:amp}\docref{xparse:amp}\bracestt{%
4977 \refcmd{collargsEnvironment}\bracestt{minipage}}\docref{xparse:+}\docref{xparse:b}}.
4978
4979 To set the raw options for all arguments, use \refcmd{CollectArgumentsRaw}, the
4980 second public command of the package. This command is exactly like
4981 \refcmd{CollectArguments}, excepts that it expects the options in the raw
4982 format and as a \emph{mandatory} argument:
4983
4984 \begin{center}
4985 \refcmd{CollectArgumentsRaw}\marg{raw options}\marg{argument specification}%
4986 \marg{next-code}\textcolor{gray}{\meta{tokens}}
4987 \end{center}
4988
4989 This leads us to the fifth way to set the environment name (an overkill, I
4990 know): \refcmd{CollectArgumentsRaw}\hyp
4991 \bracestt{\refcmd{collargsEnvironment}\bracestt{minipage}}\bracestt{+b}\marg{next-code}.
4992 Furthermore, you can use a mixture of raw and key--value options: the raw
4993 option commands include \refcmd{collargsSet}, which applies the given option
4994 keylist. The idea here (incarnated by both Auto and Memoize) is that the
4995 package will provide CollArgs with the raw options, for speed, while the
4996 author can supplement them in the friendly keylist format --- and this leads
4997 us to the sixth, and thankfully final way to set the environment name:
4998 \refcmd{CollectArgumentsRaw}\bracestt{\refcmd{collargsSet}%
4999 \bracestt{\refcollargs{environment}=minipage}}\bracestt{+b}\marg{next-code}.
5000
5001
5002 \paragraph{The initial collector}
5003
5004 As the final example, let us study Advice's initial collector; this is a macro
5005 which is used as the collector when key \refmmzauto{collector} is not given.
5006 This macro is not really \refcmd{CollectArguments}, as we sometimes state to
5007 simplify matters, but a macro which acts as the ``bridge'' between Advice and
5008 CollArgs, by compiling an invocation of \refcmd{CollectArgumentsRaw} from the
5009 given advice setup, and executing it.
5010
5011 The bridge macro is shown below in its full glory, but it is really less
5012 complicated than it might appear at first sight. In line~2, we use
5013 \refcmd{AdviceIfArgs} to see whether the argument structure of the handled
5014 command was given by the user. If it wasn't, we assume that the handled
5015 command was defined using \cs{NewDocumentCommand} or similar, and use
5016 \refcmd{GetDocumentCommandArgSpec} to retrieve it (line~3; note that
5017 \refcmd{AdviceName} holds the handled control sequence) and store it into
5018 \refcmd{AdviceArgs} (line~4), which also receives the argument specification
5019 when given by the user via key \refmmzauto{args}.
5020
5021 \tcbinputexample{_advice-CollectArgumentsRaw}[.excerpt]{%
5022 listing only, no .listing, no attachment,
5023 listing options app={numbers=left, numberstyle=\tiny, numbersep=0.5em},
5024 title={The definition of the initial collector},
5025 }
5026
5027 Lines~6--17 are somewhat of an expansion mess, because we have to construct the
5028 invocation of the CollArgs' collector from the advice setup stored in various
5029 macros. But once we think away all the (non-)expansion commands, we're left
5030 with \refcmd{CollectArgumentsRaw} plus the following three arguments:
5031 \begin{enumerate}
5032 \item The raw options (lines 8--11):
5033 \begin{enumerate}
5034 \item In line 8, the advised command's control sequence is designated as the
5035 \refcollargs{caller}. The effect is that if the given arguments don't
5036 conform to the specification, the error thrown seems to come from the
5037 advised command rather than some internal CollArgs macro. The author will
5038 be grateful for this little detail.
5039 \item In line 9, we add any \refmmzauto{raw collector options} set by Advice
5040 (plus the package deploying Advice, like Memoize); user-given options are of
5041 course possible, but not really expected here, because:
5042 \item In lines 10--11, we add the user-given \refmmzauto{collector options},
5043 if there are any, embedded under \refcmd{collargsSet}.
5044 \end{enumerate}
5045 \item The argument specification (line 14).
5046 \item The \refmmzauto{inner handler} (line 15).
5047 \end{enumerate}
5048
5049
5050
5051 \section{Reference}
5052 \label{sec:reference}
5053
5054 In this section, the extra information about keys, offered at the right of a
5055 key in parenthesis, may contain the initial value of a key, and also a default
5056 value of a key. In this context, the terms ``initial'' and ``default'' have
5057 the meaning employed by the \pkg{pgfkeys} utility (\PGFmanual{87}). The term
5058 ``initial value'' applies to the setting underlying a key (when there is no
5059 such setting, the key is marked as a ``style''), and refers to the value of
5060 this setting that is set by the package. In other contexts, we would call this
5061 the default or the package-default value, but in the \pkg{pgfkeys} parlance,
5062 the term ``default value'' applies to a key taking an argument, and refers to
5063 the value that is passed to the key in the absence of that argument.
5064 (Honestly, I only keep to this convention in the reference section; elsewhere,
5065 I often say ``default'' or ``package-default'' and mean the initial value.)
5066
5067 Another convention I keep to in this section is the color-coding of the keys
5068 and commands. Green background indicates a basic key or command, which any
5069 user might want to know about. Red background indicates other, more or less
5070 advanced keys and commands.
5071
5072
5073 \subsection{Loading and initialization}
5074 \label{sec:ref:loading}
5075 \paragraph{\hologo{LaTeX}}
5076
5077 Load Memoize by \cs{usepackage}\braces{\docaux{package}{memoize}} or
5078 \cs{usepackage}\oarg{options}\bracestt{memoize}. The latter form functions
5079 almost identically to the former followed by
5080 \refcmd{mmzset}\bracestt{\meta{options}}, with two exceptions.
5081
5082 First, when used as package options, \meta{options} may not contain the slash
5083 character (|/|), which is necessary to invoke \pkg{pgfkeys} handlers, because
5084 is it misinterpreted by \hologo{LaTeX}.\footnote{The historic \hologo{LaTeX}
5085 constraint prohibiting spaces in package options does not apply anymore.} In
5086 the rare situations requiring key handling in the package options, use option
5087 \refmmz{options}.
5088
5089 Second, key \refmmz{extract} and other extraction-related keys such as
5090 \refmmz{perl extraction options} should normally be used as a package option,
5091 or within \reffile{memoize.cfg}. Because Memoize extracts the externs while it
5092 is being loaded, executing these keys after the package is loaded will have a
5093 different effect; see the documentation of \refmmz{extract} for details.
5094
5095 Memoize extracts the externs while it is being loaded (and not, say, at the
5096 beginning of the document) because extern extraction can only be performed
5097 before the output PDF is opened, and some packages cause it to be opened when
5098 they are loaded. This also implies that Memoize must be loaded before any such
5099 package. In particular, it must be loaded before \PGF library |fadings| (see
5100 section~\ref{sec:fadings-problem}) and before the \pkg{beamer} class (see
5101 section~\ref{sec:tut:beamer}).
5102
5103 If you're familiar with the \TikZ externalization library, you might wonder
5104 whether Memoize has an equivalent of the \refcmd{tikzexternalize} command. It
5105 doesn't. Memoize assumes that if you loaded it, you want to use it --- but you
5106 can always disable it using key \refmmz{disable}, or even by loading package
5107 \refpkg{nomemoize} in its stead.
5108
5109 In \hologo{LaTeX}, initialization and finalization are completely automatic.
5110 Memoize defines several initialization and finalization styles ---
5111 \docaux{key}{begindocument/before}, \docaux{key}{begindocument},
5112 \docaux{key}{begindocument/end} and \docaux{key}{enddocument/afterlastpage}
5113 ---and executes them at the cognominal \hologo{LaTeX} hooks.
5114
5115 \paragraph{\hologo{plainTeX}}{}
5116
5117 Load Memoize by |\input memoize|. As package options cannot be provided in
5118 \hologo{plainTeX}, the author must trigger extraction from \refcmd{mmzset}
5119 using key \refmmz{extract}; I recommend doing this immediately after loading
5120 the package. This key may be invoked with or without a value. In the latter
5121 case, Memoize will extract using the package default method
5122 \refmmz{extract=perl}, unless its has been overridden from
5123 \reffile{memoize.cfg}.
5124
5125 Furthermore, as \hologo{plainTeX} has no concept of a document body, the text
5126 must be manually enclosed in \refcmd{mmzset}\bracestt{\docaux{key}{begin
5127 document}} and \refcmd{mmzset}\bracestt{\docaux{key}{end document}}; this
5128 is where the initialization and finalization hooks described above will be
5129 executed. Note that \refmmz{extract}, when used, must precede this enclosure.
5130
5131 Finally, \hologo{plainTeX} has another reason for preferring the early loading
5132 of the package. In this format, Memoize must redefine \refcmd{shipout}, at a
5133 time the meaning of this control sequence is still primitive. In particular,
5134 this means that Memoize must be loaded before \pkg{atbegshi}.
5135
5136 \begin{tcboxedraster}[raster column skip=1cm]{blankest}
5137 \tcbinputexample{plainTeX}{listing only, title=A minimal \hologo{plainTeX} example}
5138 %
5139 \tcbinputexample{ConTeXt}{listing only, title=A minimal \hologo{ConTeXt} example}
5140 \end{tcboxedraster}
5141
5142 \paragraph{\hologo{ConTeXt}}{}
5143
5144 Load Memoize by \cs{usemodule}\bracketstt{memoize} or
5145 \cs{usemodule}\bracketstt{memoize}\oarg{options}. Unlike in \hologo{LaTeX},
5146 there are no restrictions on characters allowed within \meta{options}; the
5147 remarks on key \refmmz{extract} and loading order are the same as for
5148 \hologo{LaTeX}.
5149
5150 In \hologo{LaTeX}, Memoize automatically executes its initialization and
5151 finalization code when at the beginning and the end of the document body. Due
5152 to my very limited experience with \hologo{ConTeXt}, and its project structure
5153 in particular, I don't know what the appropriate place for initialization and
5154 finalization would be in \hologo{ConTeXt}. I therefore provisionally leave it
5155 to the author to execute \refcmd{mmzset}\bracestt{\refmmz{begin document}} and
5156 \refcmd{mmzset}\bracestt{\refmmz{end document}} manually, and hope for advice
5157 on how to handle this properly.
5158
5159
5160 \paragraph{Auxiliary packages}
5161
5162 \begin{doc}{package={name=nomemoize}}
5163 Loading this package instead of Memoize completely disables memoization, but
5164 does not require removing any Memoize commands from the document (they all
5165 become no-ops). This package accepts any package options (and ignores them).
5166 \end{doc}
5167
5168 \begin{doc}{package={name=memoizable}}
5169 This package is a programmer's stub: if Memoize is loaded, it does nothing;
5170 otherwise, it provides the no-op variants Memoize commands. This package
5171 accepts no package options. Generic packages, e.g.\ \TikZ libraries, can
5172 also load this package via |\input memoizable.code.tex|. See
5173 section~\ref{sec:loading-memoize} for details.
5174 \end{doc}
5175
5176
5177 \subsection{Configuration}
5178 \label{sec:ref:configuration}
5179
5180
5181 \begin{doc}{easy,cmd={name=mmzset, par=\marg{options}}}
5182
5183 Update the Memoize configuration.
5184
5185 The \meta{options} are a comma-separated list of \meta{key}|=|\meta{value}
5186 pairs. They are processed using the \pkg{pgfkeys} utility of PGF/\TikZ (see
5187 \PGFmanual{87}), with the default path set to \docaux{key path}{mmz}.
5188
5189 The changes are local to the current \hologo{TeX} group, except for keys
5190 where explicitly noted otherwise.
5191 \end{doc}
5192
5193 \begin{doc}{easy,
5194 cmd={name=mmznext, par=\marg{options}},
5195 }
5196
5197 This command accepts the same \meta{options} as \refcmd{mmzset}, but
5198 interprets them as \emph{next-options} --- options which will be applied to
5199 the next, and only the next, \emph{auto}memoized command or environment.
5200 (Remember that a command or environment is submitted to automemoization by
5201 \refmmz{auto}|=|\marg{command or
5202 environment}\bracestt{\refmmzauto{memoize},...}; see
5203 sections~\ref{sec:tut:automemoization}, \ref{sec:tut:working-on-a-picture}
5204 and \ref{sec:ref:advicememoization} for details.)
5205
5206 Remarks for the author:
5207
5208 \begin{itemize}
5209 \item Key \refmmz{enable} has no effect inside \refcmd{mmznext}.
5210 \item If \refcmd{mmznext} is used more than once preceding an automemoized
5211 command, only the final invocation takes effect.
5212 \item The next-options also apply to commands and environments for which
5213 memoization is autodisabled via \refmmz{auto}|=|\marg{command or
5214 environment}\bracestt{\refmmzauto{nomemoize},...}.
5215 \item It is safe to set the next-options in front of a command submitted to
5216 automemoization which does not actually undergo memoization in this
5217 particular instance. In other words, the absence of memoization will not
5218 cause the next-options to ``leak'' to the next automemoized command.
5219 \item Only the linear (execution) order of \refcmd{mmznext} and the
5220 automemoized command matters. Key \refcmd{mmznext} will correctly apply to
5221 a single following automemoized command even if it occurs outside the group
5222 which that command is executed from; and it will apply to the following
5223 automemoized command even if it is called within a group closed before that
5224 command is executed.
5225 \end{itemize}
5226
5227 Remarks for the programmer:
5228 \begin{itemize}
5229 \item The next-options are set globally.
5230 \item The effect of \refcmd{mmznext} is \emph{not} cumulative. Consequently,
5231 \refcmd{mmznext}\braces{} clears the next-options.
5232 \item The next-options are applied by executing \refcmd{mmzAutoInit} within
5233 the advice. Any piece of advice applying the next options should also clear them
5234 when the \refmmzauto{run conditions} are not met. This is streamlined by
5235 style \refmmzauto{apply options}, intended for use within \refmmz{auto}
5236 declarations. Out of the box, this style is deployed by
5237 \refmmzauto{memoize}, \refmmzauto{nomemoize} and \refmmzauto{noop}, but it
5238 may be used by any piece of advice. Note that the \refmmzauto{outer handler}
5239 declared by this style opens a group (to apply the options in) but leaves
5240 it to the (undeclared) \refmmzauto{inner handler} to close that group.
5241 \item Key \refmmz{enable} has no effect inside \refcmd{mmznext} because when
5242 Memoize is disabled upon encountering an automemoized command, the advice
5243 bails out without ever applying the next-options. More generally, this
5244 applies to any advised command whose run conditions require
5245 \refcmd{ifmemoize} to be true. Key \refmmz{disable}, on the other hand,
5246 takes effect, because \refcmd{ifmemoize} is checked within \refcmd{Memoize}
5247 as well.
5248 \end{itemize}
5249 \end{doc}
5250
5251 \begin{doc}{easy,file={name=memoize.cfg, desc=file}}
5252 The configuration file, loaded just before processing the package options.
5253 It will typically contain a \refcmd{mmzset} command, but it may contain any
5254 \hologo{TeX} code.
5255
5256 As for any other file loaded by \hologo{TeX}, the location of the file
5257 determines whether it applies system-wide, user-wide or directory-wide.
5258
5259 This file is also loaded by package \refpkg{nomemoize}, on the off chance it
5260 defines some commands other than \refkeypath{/mmz} keys. (It is not loaded
5261 by package \refpkg{memoizable}, though).
5262 \end{doc}
5263
5264 \begin{doc}{key={name=options, par=\marg{options}}}
5265 Execute \meta{options} as if they were given an an argument to \refcmd{mmzset}.
5266
5267 This option primarily exists to allow any key--value pair accepted by
5268 \refcmd{mmzset} to be used as a package option. In particular, this applies
5269 to key handlers (see \PGFmanual{87}), because their invocation includes a slash
5270 (|/|). For example, one cannot directly use
5271 \begin{center}
5272 \code{\cs{usepackage}[\refmmz{perl extraction
5273 options}/.prefix=\braces{\braces{\refscript{memoize-extract.pl--quiet}}} ]\braces{memoize}}
5274 \end{center}
5275 to add (prepend) option \refscript{memoize-extract.pl--quiet} to the
5276 invocation of the perl extraction script \refscript{memoize-extract.pl}, but
5277 this will work:
5278 \begin{center}
5279 \code{\cs{usepackage}[\refmmz{options}=\braces{\refmmz{perl extraction
5280 options}/.prefix=\braces{\braces{\refscript{memoize-extract.pl--quiet}}} ]\braces{memoize}}}
5281 \end{center}
5282 \end{doc}
5283
5284 \begin{doc}{cmd={name=nommzkeys, par=\marg{key}}}
5285 In package \refpkg{memoize}, this command is a no-op; in packages
5286 \refpkg{nomemoize} and \refpkg{memoizable}, it defines key
5287 \refkeypath{/mmz}|/|\meta{key} as a no-op.
5288
5289 As explained in section~\ref{sec:tut:final}, use this command to declare any
5290 \refkeypath{/mmz} keys you have used outside \refcmd{mmzset} when switching
5291 to package \refpkg{nomemoize}.
5292 \end{doc}
5293
5294
5295 \subsection{Memoization}
5296 \label{sec:ref:memoization}
5297
5298 \subsubsection{Manual memoization commands}
5299 \label{sec:ref:memoization:manual}
5300
5301 \begin{doc}{easy,
5302 cmd={name=mmz, par=\oarg{options}\marg{code}},
5303 env={name=memoize, par=\oarg{options}},
5304 }
5305 Submit \meta{code} or \meta{environment body} to memoization.
5306
5307 Prior to memoization, the configuration is locally updated by executing
5308 \meta{options} given as the optional argument to this command, i.e.\ the
5309 given options take precedence to options previously set via \refcmd{mmzset}.
5310 Note that next-options, set by \refcmd{mmznext}, are not applied.
5311
5312 The effect of the macro and of the environment version of this command is the
5313 same, except that the command version memoizes \meta{code} exactly as-is,
5314 while the environment version trims away any spaces at the beginning and the
5315 end of the code.\footnote{What actually happens is that at the beginning of
5316 the environment body, all space tokens will be discarded. At the end of
5317 the body, no spaces are actually discarded; Memoize simply issues an
5318 \refcmd{unskip}. This should not matter to a regular user who simply
5319 writes down the environment.} The space-trimming feature of the environment
5320 ensures that you can write |\begin{memoize}| and |\end{memoize}| in separate
5321 lines (as shown above), but no extra space will creep into the extern.
5322
5323 The space-trimming feature of the environment, which trims spaces at the
5324 beginning and at the end of the \meta{environment body}, should not be
5325 confused with the effect of \refmmz{ignore spaces}, which ignores spaces
5326 following the environment end-tag (in \hologo{LaTeX}, |\end{...}|) --- and
5327 which does not apply to manual memoization at all!
5328
5329 The argument of \refcmd{mmz} \emph{must} be enclosed in braces.
5330 \end{doc}
5331
5332 \begin{doc}{easy,
5333 cmd={name=nommz, par=\oarg{options}\marg{code}},
5334 env={name=nomemoize, par=\oarg{options}},
5335 }
5336 Disable Memoize for the span of the compilation of \meta{code} or
5337 \meta{environment body}.
5338
5339 This command consumes the \meta{options} in the same way as
5340 \refcmd{mmz}\slash \refenv{memoize} described above. The macro and the
5341 environment version of the command exhibit the same space-trimming behaviour
5342 as their \refcmd{mmz}\slash \refenv{memoize} counterparts, and the argument
5343 of \refcmd{nommz} \emph{must} be enclosed in braces.
5344 \end{doc}
5345
5346
5347 \subsubsection{Basic configuration}
5348 \label{sec:ref:memoization:basic}
5349
5350 \begin{doc}{easy,
5351 key={name=enable},
5352 key={name=disable},
5353 }
5354 Enable or disable the functionality of the package.
5355
5356 What happens when Memoize is enabled depends on the memoization mode
5357 (\refmmz{normal}, \refmmz{readonly}, \refmmz{recompile}) and many other
5358 factors. When the package is disabled, it neither creates new memos and
5359 externs, nor uses the existing ones; this applies to both manual and
5360 automatic memoization. The effect is close to not having Memoize loaded, or
5361 to loading NoMemoize, but it is not completely the same; for example, the
5362 record file (\dmmz) \emph{is} updated while Memoize is disabled, reflecting
5363 the fact that nothing was memoized (or utilized) in the disabled
5364 state.\footnote{Cleaning the folder (\S\ref{sec:cleanup-scripts}) after
5365 disabling the package for the entire document is thus a bad idea.}
5366
5367 If these keys are used in the preamble, their effect is delayed until the
5368 beginning of the document, to ensure that Memoize is never enabled in the
5369 preamble. Other than that, all these keys do is set the \hologo{TeX}
5370 conditional \refcmd{ifmemoize}, which you may use in your code to test
5371 whether Memoize is enabled. You may also use \refcmd{memoizetrue} and
5372 \refcmd{memoizefalse}, as long as you never enable the package in the
5373 preamble.
5374
5375 Key \refmmz{enable} cannot be applied to automemoized commands via
5376 \refcmd{mmznext}. It will take effect for manual memoization, though, and
5377 key \refmmz{disable} will work for both, as expected.
5378 \end{doc}
5379
5380 \begin{doc}{easy,
5381 key={name=normal, desc=the initial mode},
5382 key={name=readonly},
5383 key={name=recompile},
5384 }
5385
5386 Select the memoization mode.
5387
5388 Each piece of code submitted to (either manual or automatic) memoization is
5389 associated to several files: one c-memo, one cc-memo, and some externs (zero
5390 or more, typically one). When Memoize encounters a piece of code submitted
5391 to memoization, it takes one of the following actions:
5392
5393 \begin{description}
5394 \item[memoization] The code is compiled in a special way which produces the
5395 associated memos and externs.
5396 \item[utilization] The code is not compiled. Its effect is replicated by
5397 processing the cc-memo; typically, this includes the single extern into the
5398 document.
5399 \item[regular compilation] The code is compiled as if Memoize was absent or
5400 disabled (the memos and externs are neither utilized nor produced).
5401 \end{description}
5402
5403 \begin{minipage}[t]{0.48\linewidth}
5404 The action taken depends on the memoization mode and on whether all the
5405 memos and externs associated with the code exist, as shown in the table on
5406 the right. Note that a single missing memo or extern implies the ``no''
5407 column of the table, and that memoization will create \emph{all} the
5408 associated memos and externs, even those which already exist.
5409 \end{minipage}
5410 \hfill
5411 \begin{tabular}[t]{@{}lll@{}}
5412 \toprule
5413 &\multicolumn{2}{c}{Do the memos and externs exist?}\\
5414 \cmidrule{2-3}
5415 mode&yes&no\\
5416 \midrule
5417 normal&utilization&memoization\\
5418 readonly&utilization&regular compilation\\
5419 recompile&memoization&memoization\\
5420 \bottomrule
5421 \end{tabular}
5422
5423 The memoization mode only has effect when Memoize is \refmmz{enable}d. Mode
5424 selection is orthogonal to enabling\slash disabling the package; for example,
5425 if you switch to a new mode while the package is disabled, the new mode will
5426 be in effect once the package is enabled.
5427 \end{doc}
5428
5429
5430 \begin{doc}{easy,
5431 desc={style},
5432 key={name=verbatim},
5433 key={name=verb},
5434 key={name=no verbatim, desc={style, the initial mode}},
5435 }
5436 When \refmmz{verbatim} or \refmmz{verb} is in effect, the code submitted to
5437 memoization is read verbatim; \refmmz{no verbatim} reverts to the normal,
5438 non-verbatim collection of the code. This applies to both manual and
5439 automatic memoization.
5440
5441 The long version, \refmmz{verbatim}, switches to the full verbatim mode,
5442 where all characters are assigned category code 12 (other). With the short
5443 version, \refmmz{verb}, the braces, |{| and |}|, retain category codes 1 and
5444 2, which can be useful for verbatim collection of optional arguments. For
5445 details, see the documentation of CollArgs' \refcollargs{verbatim} in
5446 section~\ref{sec:ref:collargs}.
5447
5448 Under the hood, these keys have two effects. First, they are passed on to the
5449 argument collector (typically, \refcmd{CollectArguments} of the auxiliary
5450 package CollArgs; for details, see section~\ref{sec:ref:advicememoization}),
5451 instructing it to collect the code in the specified fashion, as described
5452 above. Second, if the collected verbatim code is eventually compiled (either
5453 regularly, or memoized), Memoize first rescans it using \refcmd{scantokens}.
5454 \end{doc}
5455
5456 \begin{doc}{easy,
5457 key={name=padding left, par=\meta{dimension},
5458 desc={no default, initially |1| |in|}},
5459 key={name=padding right, par=\meta{dimension},
5460 desc={no default, initially |1| |in|}},
5461 key={name=padding top, par=\meta{dimension},
5462 desc={no default, initially |1| |in|}},
5463 key={name=padding bottom, par=\meta{dimension},
5464 desc={no default, initially |1| |in|}},
5465 }
5466 Set the left/right/top/bottom padding of the extern in the extern PDF.
5467
5468 Without padding, the (PDF) page holding the extern would tightly fit the
5469 bounding box of the extern. These keys enlarge the extern page by the given
5470 amounts, so that any parts of the extern lying outside the bounding box will
5471 be correctly included when using the extern. See
5472 section~\ref{sec:tut:padding} for details.
5473
5474 \meta{dimension} is evaluated with \hologo{eTeX}'s |\dimexpr|, and may
5475 contain control sequences \docaux{cmd}{width}, \docaux{cmd}{height} and
5476 \docaux{cmd}{depth}, which will refer to the dimensions of the extern.
5477 \refcmd{width} and friends behave like dimension registers, so it is ok to write
5478 e.g.\ \refmmz{padding right}|=0.5\width|.
5479
5480 The default padding is what \hologo{pdfTeX} puts around the page anyway, 1
5481 inch, but we use |1 in| rather than |1 true in|, which is the true default
5482 value of PDF registers \docref{pdfvar:horigin} and \docref{pdfvar:vorigin},
5483 as we want the padding to adjust with \refcmd{mag}nification.
5484 \end{doc}
5485
5486 \begin{doc}{easy,key={name=padding, par=\meta{dimension}, desc={style, no default}}}
5487 Set all the above |padding| keys to the given value.
5488 \end{doc}
5489
5490 \begin{doc}{
5491 easy,
5492 key={name=context, par=\meta{tokens}, desc={cumulative, no default,
5493 initially set by \refmmz{padding to context}}},
5494 key={name=clear context},
5495 }
5496 These keys append the given \meta{tokens} to the \emph{context expression},
5497 or clear this expression. Memoized code gets recompiled whenever the
5498 expansion of the context expression changes.
5499
5500 The \meta{tokens} must be fully expandable (modulo protection); in
5501 \hologo{LaTeX}, they will be expanded by |\protected@edef| when calculating
5502 the md5sum of the context.
5503
5504 The context expression is evaluated at the end of the memoization, and at
5505 utilization attempts after inputting the c-memo (note that the c-memo
5506 contains any additions to the context expression accumulated during
5507 memoization). At evaluation, the given context expression is fully expanded,
5508 yielding the \emph{context value}, whose md5 sum forms the \meta{context md5
5509 sum} part of the filename of the cc-memo and the extern.
5510
5511 These keys may be used both prior to the memoization process, or during
5512 memoization. In the former case, their effect is local to the current group;
5513 in the latter case, the effect is global, so that the changes surely survive
5514 until the end of memoization, when the c-memo, where the context expression
5515 is stored, is written into a file.
5516
5517 Under the hood, these keys manipulate token registers
5518 \docaux{cmd}{mmzContext} and \docaux{cmd}{mmzContextExtra}, changing the
5519 contents of the former while not memoizing, and the contents of the latter
5520 during memoization. These token registers may also be manipulated directly
5521 by the user, as long as one keeps to the convention of adjusting
5522 \refcmd{mmzContext} locally and only while not memoizing, and adjusting
5523 \refcmd{mmzContextExtra} globally and only during memoization.
5524 \end{doc}
5525
5526 \begin{doc}{
5527 key={easy, name=meaning to context,
5528 par=\marg{comma-separated list of commands and environment names}},
5529 key={name=csname meaning to context, par=\marg{control sequence name}},
5530 key={name=key meaning to context,
5531 par=\marg{full path to a pgfkeys command key}},
5532 key={name=key value to context,
5533 par=\marg{full path to a pgfkeys value key}},
5534 }
5535 These keys append the definition of the given construct to the context.
5536
5537 Essentially, the ``meaning'' keys append \cs{meaning}\meta{control sequence}
5538 to the context, for the appropriate \meta{control sequence}. For example,
5539 \refmmz{meaning to context} appends \cs{meaning}\cs{foo} when given \cs{foo}
5540 as in item in the list, and it appends the internal environment macros
5541 appropriate to the format when given an environment name. Similarly,
5542 \refmmz{key meaning to context} resolves \meta{control sequence} to the
5543 internal macro holding the key command.
5544
5545 Key \refmmz{key value to context} should be used on keys which store values,
5546 e.g.\ keys initialized by \pkg{pgfkeys} handler \code{.initial}; see
5547 \PGFmanual{87.3.4 and \S87.4.5}.
5548
5549 All the keys prefix the meaning\slash value by the name of the command\slash
5550 environment\slash key, in order to prevent ambiguous contexts, see
5551 section~\ref{sec:cross-referencing} for details. Furthermore, they all
5552 operate through \code{\cs{csname}...\cs{endcsname}} construct, allowing one
5553 to safely add internal commands to the context.
5554 \end{doc}
5555
5556 \begin{doc}{
5557 keypath={/handlers}, desc=handler,
5558 key={easy, name=.meaning to context},
5559 key={name=.value to context},
5560 }
5561 These are the handler variants of \refmmz{key meaning to context} and
5562 \refmmz{key value to context}.
5563 \end{doc}
5564
5565
5566 \begin{doc}{key={name=padding to context,desc=style}}
5567 This key appends the values of the \refmmz{padding} keys to the context,
5568 causing the memoized code to be recompiled whenever the padding values are
5569 changed. This key is used to initialize \refmmz{context}, so the author
5570 normally shouldn't have to use this key.
5571 \end{doc}
5572
5573 \begin{doc}{
5574 key={name=salt, par=\meta{tokens}, desc={cumulative, no default,
5575 initially empty}},
5576 key={name=clear salt},
5577 key={name=meaning to salt,
5578 par=\marg{comma-separated list of commands and environment names}},
5579 key={name=csname meaning to salt, par=\marg{control sequence name}},
5580 key={name=key meaning to salt,
5581 par=\marg{full path to a pgfkeys command key}},
5582 key={name=key value to salt,
5583 par=\marg{full path to a pgfkeys value key}},
5584 keypath={/handlers}, desc=handler,
5585 key={name=.meaning to salt},
5586 key={name=.value to salt},
5587 }
5588 These keys manipulate the \emph{salt} expression, internally stored in token
5589 register \docaux{cmd}{mmzSalt}. Fully expanded (at the beginning of
5590 memoization), this expression is prepended to the memoized code and
5591 contributes to its code md5 sum. In other words, two pieces of the same code
5592 memoized under a different expansion of the salt will behave as if they
5593 contained different code; most importantly, they will have distinct c-memos.
5594
5595 The keys listed above behave in the same way as their \refmmz{context}
5596 counterparts used outside memoization, i.e.\ in the same way as the local
5597 variant of the \refmmz{context} keys. Using the salt keys during memoization
5598 has no effect.
5599
5600 While salt and context are mostly interchangeable, one example where only the
5601 salt can do the job is presented in the Beamer section~\ref{sec:per-overlay}.
5602 \end{doc}
5603
5604 \begin{doc}{
5605 cmd={name=mmzNoRef, par=\marg{reference key}, desc=\hologo{LaTeX} only},
5606 cmd={name=mmzForceNoRef, par=\marg{reference key}, desc=\hologo{LaTeX} only},
5607 }
5608 These macros append the current value of the \meta{reference key} to the
5609 context,\footnote{More precisely, it is \meta{reference key}|=|\marg{current
5610 value} which is appended.} causing the memoized code to be recompiled
5611 when the reference changes.
5612
5613 If the reference key is undefined, \refcmd{mmzNoRef} aborts memoization,
5614 while \refcmd{mmzForceNoRef} uses \cs{relax} as the reference string.
5615
5616 These commands are deployed in the implementation of \refmmzauto[show
5617 keypath]{ref}, \refmmzauto{force ref} and friends. If the
5618 cross-referencing commands you are using are advised by these keys, you most
5619 likely have no need of these macros.
5620 \end{doc}
5621
5622 \begin{doc}{
5623 key={name=capture, par=\docAux{of=key:/mmz/capture, value={name=hbox}, text=\alt,
5624 value={name=vbox,of=key:/mmz/capture}},
5625 desc={no default, initially \refmmz{capture=hbox}}}
5626 }
5627 Select the capture mode. This setting only applies to the default
5628 memoization \refmmz{driver}.
5629
5630 By default, it is assumed that the memoized code should be executed in the
5631 horizontal mode, so the default memoization driver captures the output of the
5632 memoized code in a \refcmd{hbox} --- and also issues a \refcmd{quitvmode}
5633 (both in the document and in the cc-memo), just in case the memoized code
5634 occurs at the start of the paragraph.
5635
5636 Use \refmmz{capture}|=|\refmmz{capture=vbox} to execute the memoized code in
5637 the vertical mode: Memoize will capture the output of the memoized code in a
5638 \refcmd{vbox}, and avoid issuing \refcmd{quitvmode}. For example, this
5639 capture mode is necessary to memoize a \refenv{verbatim} environment.
5640 \end{doc}
5641
5642
5643 \subsubsection{Inside the memoization process}
5644 \label{sec:ref:memoization:process}
5645
5646 \begin{doc}{easy, cmd={name=mmzAbort}}
5647 This command aborts the ongoing memoization.
5648
5649 The memoization will proceed as usual (i.e.\ the extern boxes and the cc-memo
5650 code will be produced), but at the end of this process, no memos will be
5651 produced, no externs shipped out to extern pages, and no record files
5652 updated.
5653 \end{doc}
5654
5655 \begin{doc}{easy, cmd={name=mmzUnmemoizable}}
5656 This command aborts the ongoing memoization, and marks the submitted code as
5657 unmemoizable.
5658
5659 The ongoing memoization produces a c-memo setting conditional
5660 \docaux{cmd}{ifmmzUnmemoizable} to true. Upon utilizing this c-memo, the
5661 system switches to regular compilation.
5662
5663 For example, if you are automemoizing |tcolorbox|es of package
5664 \pkg{tcolorbox}, you will want to refrain from memoizing boxes marked as
5665 |breakable| or |float|ing. Simply aborting the memoization cannot do the
5666 trick here, as memoization compiles the submitted code in a \hologo{TeX} box.
5667 Marking a |breakable| or |float|ing |tcolorbox| as unmemoizable (either
5668 manually using this macro, or automatically using \refmmz{auto key}) makes
5669 sure that after the first compilation when memoization is attempted, the box
5670 will be compiled regularly, and will have the intended ability to break
5671 across pages, or float.
5672 \end{doc}
5673
5674
5675 \begin{doc}{
5676 cmd={name=Memoize, par=\marg{key}\marg{code}},
5677 }
5678 Depending on various factors, this command either memoizes \meta{code} under
5679 key \meta{key}, utilizes the results of a previous memoization, or performs a
5680 regular compilation of \meta{code}.
5681
5682 The outcome of executing this command --- memoization, utilization or regular
5683 compilation --- depends upon the Memoize's state (\refcmd{ifmemoize},
5684 \refcmd{ifmemoizing}) and mode (\refmmz{normal}, \refmmz{readonly},
5685 \refmmz{recompile}), and the existence of the relevant memos and externs.
5686 The decision process is depicted in section~\ref{sec:Memoize}.
5687
5688 This command expects to be executed in a dedicated group, which it will close
5689 itself.
5690
5691 Invoking memoization through \refcmd{Memoize} might be useful for packages
5692 which want to save the results of intensive computations, regardless of
5693 whether the author loads (and enables) memoization or not. However, this
5694 usage is not yet officially allowed, because there is currently no way to
5695 load the core memoization routines without loading the entire package,
5696 thereby forcing the author to use Memoize.
5697 \end{doc}
5698
5699 \begin{doc}{
5700 key={
5701 name=driver, par=\marg{code},
5702 desc={no default, initially \refcmd{mmzSingleExternDriver}}
5703 },
5704 }
5705 This key sets \meta{code} as the memoization driver.
5706
5707 Given some code submitted to memoization, the memoization driver should
5708 produce the memos and externs which will replicate the effect of that code
5709 (while retaining its regular effect). For details, see
5710 section~\ref{sec:memoization-drivers}.
5711
5712 Typically, the \meta{code} argument of this key will consist of a single
5713 control sequence (the driver control sequence), but any amount of tokens is
5714 allowed. Memoize executes the driver followed by the code which it is
5715 supposed to memoize, in braces, and only cares that the driver consumes that
5716 code.
5717 \end{doc}
5718
5719
5720 \begin{doc}{
5721 key={name=at begin memoization, par=\meta{code}, desc={cumulative, initially empty}},
5722 key={name=at end memoization, par=\meta{code}, desc={cumulative, initially empty}},
5723 key={name=after memoization, par=\meta{code}, desc={cumulative, initially empty}},
5724 }
5725 Use these keys to set up memoization hooks.
5726
5727 These keys may be used both prior to the memoization process, or during
5728 memoization. In the former case, their effect is local to the current group;
5729 in the latter case, the code given to \refmmz{at begin memoization} is
5730 executed immediately, while the assignment performed by the other two keys is
5731 global, so that the changes surely survive until the end of memoization.
5732
5733 The code given to hook \refmmz{at begin memoization} is kept in macro
5734 \docaux{cmd}{mmzAtBeginMemoization}, while the content of the other two hooks
5735 resides in two macros per hook: \docaux{cmd}{mmzAtEndMemoization} and
5736 \docaux{cmd}{mmzAtEndMemoizationExtra}, and \docaux{cmd}{mmzAfterMemoization}
5737 and \docaux{cmd}{mmzAfterMemoizationExtra}. All these macros may be
5738 manipulated directly by the user,\footnote{Use \cs{appto}, \cs{eappto},
5739 \cs{gappto} and \cs{xappto} of package \pkg{etoolbox} (loaded by Memoize)
5740 to easily append code to these macros.} as long as one keeps to the
5741 convention of adjusting the macros without ``\texttt{Extra}'' locally and
5742 only while not memoizing, and adjusting the macros with ``\texttt{Extra}''
5743 globally and only during memoization. The ``\texttt{Extra}'' macros require
5744 global assignments as they might be manipulated by code residing within a
5745 \hologo{TeX} group of any depth.
5746
5747 These complications explained, let us take a look at how memoization proceeds
5748 to learn when the hooks are used:
5749 \begin{enumerate}
5750 \item Initialize various conditionals, macros and token registers. (Here is
5751 where the ``\texttt{Extra}'' hooks are cleared.) Remember that at this
5752 point, we're inside a group opened by \refcmd{Memoize}.
5753 \item Execute \refmmz{at begin memoization} hook, i.e.\ the contents of
5754 macro \refcmd{mmzAtBeginMemoization}.
5755 \item Execute the memoization \refmmz{driver}.
5756 \item Execute \refmmz{at end memoization} hook, i.e.\ the contents of macros
5757 \refcmd{mmzAtEndMemoization} and \refcmd{mmzAtEndMemoizationExtra} (in this
5758 order).
5759 \item Write out the memos and ship out the externs to extern pages (unless
5760 memoization was aborted).
5761 \item Close the memoization group.
5762 \item Execute \refmmz{after memoization} hook, i.e.\ the contents of
5763 macros \refcmd{mmzAfterMemoization} and \refcmd{mmzAfterMemoizationExtra}
5764 (in this order).
5765 \end{enumerate}
5766 \end{doc}
5767
5768 \begin{doc}{
5769 cmd={name=mmzCMemo, desc={token register, global, empty at the start of memoization}},
5770 }
5771 This token register mediates the construction of the c-memo. During
5772 memoization (and only during memoization), arbitrary code may be added to
5773 this register; at the end of memoization, Memoize writes out its contents to
5774 the free-form part of the c-memo.
5775
5776 All assignments to this register should be global. Use \refcmd{gtoksapp} and
5777 \refcmd{xtoksapp} to easily append tokens to the register.
5778 \end{doc}
5779
5780
5781 \begin{doc}{
5782 cmd={name=mmzCCMemo, desc={token register, global, empty at the start of memoization}},
5783 }
5784 This token register mediates the construction of the cc-memo. During
5785 memoization, the memoization driver should append cc-memo code to
5786 \refcmd{mmzCCMemo}; at the end of memoization, Memoize writes out its
5787 contents to the cc-memo (preceded by the list of produced externs).
5788
5789 All assignments to this register should be global. Local assignments would
5790 not work, because the memoized code may contain commands, like \cs{label} and
5791 \cs{ref}, which contribute content to cc-memo as well, but these commands may
5792 appear within a \hologo{TeX} group of any depth.
5793
5794 Use \refcmd{gtoksapp} and \refcmd{xtoksapp} to easily append tokens to the
5795 register.
5796 \end{doc}
5797
5798 \begin{doc}{
5799 par=\meta{token register}\marg{tokens},
5800 cmd={name=toksapp},
5801 cmd={name=gtoksapp},
5802 cmd={name=etoksapp},
5803 cmd={name=xtoksapp},
5804 }
5805 These commands append the given \meta{tokens} to the \meta{token register}.
5806 \refcmd{etoksapp} and \refcmd{xtoksapp} expand the \meta{tokens} before
5807 appending them; \refcmd{gtoksapp} and \refcmd{xtoksapp} perform a global
5808 assignment.
5809
5810 These commands are actually provided by CollArgs, and they are defined only
5811 if they don't already exist; in particular, note that \hologo{LuaTeX}
5812 provides them as primitives.
5813
5814 Unlike the \hologo{LuaTeX} primitive variant, these commands require the
5815 \meta{token register} to be given by a (|\toksdef|fed) control sequence; it
5816 cannot be given as |\toks|\meta{number}.
5817 \end{doc}
5818
5819 \begin{doc}{
5820 cmd={name=ifmemoize},
5821 cmd={name=memoizetrue},
5822 cmd={name=memoizefalse},
5823 }
5824 Use the \hologo{TeX}-style conditional \refcmd{ifmemoize} to test whether
5825 Memoize is currently enabled. Within the document body, the conditional may
5826 be set using \refcmd{memoizetrue} and \refcmd{memoizefalse}, which are then
5827 functionally equivalent to \refmmz{enable} and \refmmz{disable}. Do not set
5828 the conditional in the preamble of the document (unless you really know what
5829 you are doing).
5830 \end{doc}
5831
5832 \begin{doc}{
5833 cmd={name=ifmemoizing,desc={readonly}},
5834 }
5835 Use this \hologo{TeX}-style conditional to test whether Memoize is currently
5836 memoizing. It may be only inspected; you should \emph{never} set this
5837 conditional yourself.
5838 \end{doc}
5839
5840 \begin{doc}{
5841 cmd={name=ifinmemoize,desc={readonly}},
5842 }
5843 Use this \hologo{TeX}-style conditional to test whether Memoize is currently
5844 active, in the sense of either memoizing or regularly compiling some code ---
5845 so inside a call to \refcmd{Memoize}. The conditional may be only inspected;
5846 you should \emph{never} set it yourself.
5847 \end{doc}
5848
5849 \begin{doc}{
5850 cmd={name=mmzSingleExternDriver, par=\marg{code}},
5851 }
5852 This is the default memoization \refmmz{driver}, producing exactly one extern
5853 containing whatever is typeset by the submitted \meta{code}.
5854
5855 The \meta{code} is compiled either within a horizontal or vertical box,
5856 depending on the value of key \refmmz{capture}. In the case of a horizontal
5857 capture, the driver makes sure that the horizontal mode is entered prior to
5858 both typesetting the resulting box in the document, or utilizing the extern.
5859
5860 For the implementational details, see
5861 section~\ref{sec:memoization-driver-default}.
5862 \end{doc}
5863
5864 \begin{doc}{
5865 cmd={name=mmzExternalizeBox, par=\marg{box}\marg{token register}},
5866 }
5867 This macro is intended to be called by memoization drivers to produce an
5868 extern page. The given \meta{box} is dumped into the document as a separate
5869 extern page, while the \meta{token register} receives the cc-memo extern
5870 inclusion code.
5871
5872 The \meta{box} may be given either as a control sequence (declared via
5873 |\newbox|), or as box number. The resulting extern page will contain a
5874 \emph{copy} of the given box, padded by the \refmmz{padding} values in effect
5875 at the time of invocation of \refcmd{mmzExternalizeBox}.
5876
5877 An implementation detail is that \refcmd{mmzExternalizeBox} does not ship out
5878 the extern page immediately. This action is delayed until the end of the
5879 memoization process; more precisely, it is carried out (in tandem with
5880 writing out the c-memo and the cc-memo) between execution of hooks \refmmz{at
5881 end memoization} and \refmmz{after memoization}. This delay guarantees
5882 that no extern pages are produced in the event of aborting memoization, even
5883 if the abortion is triggered after executing \refcmd{mmzExternalizeBox}.
5884
5885 The \meta{token register} may be given either as a control sequence (declared
5886 via |\newtoks|) or as control sequence |\toks| followed by the register
5887 number. The register will receive the code, which, when executed from the
5888 cc-memo, includes the extern file into the main document. This code consists
5889 of a single invocation of \refcmd{mmzIncludeExtern}. It is the
5890 responsibility of the driver to include the code received by \meta{token
5891 register} in the register \refcmd{mmzMemo}, whose contents are, unless
5892 memoization is aborted, written into the cc-memo. (See
5893 \refcmd{ifmmzkeepexterns} and \refmmz{after memoization} to learn about
5894 another way to use the code received by \meta{token register}.)
5895
5896 The invocation of \refcmd{mmzIncludeExtern} in the produced extern-inclusion
5897 code is adapted to the type of the box (horizontal or vertical), which is
5898 detected automatically --- the memoization driver does not need to inform
5899 \refcmd{mmzExternalizeBox} about this type explicitly.\footnote{This does not
5900 negate the need for key \refmmz{capture}, which applies to the default ---
5901 and therefore generic --- memoization driver. This driver cannot know
5902 whether the memoized code would prefer to be compiled in a horizontal or
5903 vertical box. It is precisely key \refmmz{capture} which gives the user an
5904 opportunity to inform Memoize about this preference. Only once the
5905 memoized code is compiled into a box of the appropriate type, it is trivial
5906 to detect the type of that box.}
5907 \end{doc}
5908
5909 \begin{doc}{
5910 cmd={name=ifmmzkeepexterns, desc={initially \cs{iffalse}}},
5911 cmd={head prefix=\texttt{\string\global}, name=mmzkeepexternstrue},
5912 cmd={head prefix=\texttt{\string\global}, name=mmzkeepexternsfalse},
5913 }
5914 Setting this conditional to true makes Memoize keep the extern boxes in the
5915 global temporary storage even after shipping them out as extern pages. (The
5916 temporary storage is emptied at the start of the next memoization.)
5917
5918 The extern inclusion code received by the \refcmd{mmzCCMemo} when executing
5919 \refcmd{mmzExternalizeBox} is primarily meant to be executed by inputting the
5920 cc-memo file; i.e.\ when the cc-memo is input, \refcmd{mmzIncludeExtern} is
5921 defined to include the extern file into the document. However, it sometimes
5922 makes sense to execute the cc-memo contents immediately after memoization;
5923 for example, if memoization produces several externs, intricately integrated
5924 into the surrounding environment, it might be cumbersome to replicate their
5925 typesetting both in the memoizing compilation and in the cc-memo code ---
5926 easier to build up the cc-memo code and execute it right after memoization.
5927 This is why Memoize, just before executing the contents of \refmmz{after
5928 memoization} hook, redefines \refcmd{mmzIncludeExtern} to include externs
5929 from the temporary storage rather than from (at that point still
5930 non-existing) extern files. However, as this mechanism requires Memoize to
5931 keep the externs around even after memoization, it is not enabled by default:
5932 it must be enabled by (globally) setting conditional
5933 \refcmd{ifmmzkeepexterns} to true.
5934 \end{doc}
5935
5936 \begin{doc}{
5937 key={keypath=/mmz/auto, name=integrated driver, par=\marg{name}, desc=style}
5938 }
5939 Use this key to easily setup a memoization driver which is integrated into
5940 the command itself.
5941
5942 \begin{tcolorbox}[warning]
5943 This is an auto-key residing in keypath \refkeypath{/mmz/auto}.
5944 \end{tcolorbox}
5945
5946 An integrated driver must have a way of telling whether it is memoizing or
5947 regularly compiling the code. This key declares a driver-specific
5948 conditional which may be inspected, using \refcmd{IfMemoizing}, to determine
5949 this. The conditional is set to true by the formal \refmmz{driver} of the
5950 command (set up by the invocation of this key), executed at the start of
5951 memoization; it should never be set elsewhere. See
5952 section~\ref{sec:memoization-multiple-externs} for details and an example.
5953 \end{doc}
5954
5955 \begin{doc}{
5956 cmd={name=IfMemoizing, par=\oarg{offset}\marg{name}\marg{true code}\marg{false code}}
5957 }
5958 This \hologo{LaTeX}-style conditional is meant to be used by the
5959 \refmmzauto{integrated driver} with the given \meta{name}. It tests whether
5960 this particular driver is currently memoizing some code.
5961
5962 Potentially recursive commands are supported via the optional argument
5963 \meta{offset}. If given, the conditional will only execute the \meta{true
5964 code} when the current \hologo{TeX} group level matches the \hologo{TeX}
5965 group level at the time of the invocation of the formal driver (held in
5966 \refcmd{memoizinggrouplevel}), plus the \meta{offset}. In effect, the inner
5967 invocation of the integrated driver will perform a regular compilation. For
5968 details, see section~\ref{sec:memoization-complex-single-driver}.
5969 \end{doc}
5970
5971 \begin{doc}{
5972 cmd={name=memoizinggrouplevel,desc={readonly}},
5973 }
5974 During memoization, this macro holds the \hologo{TeX} group level in effect
5975 at the start of the memoization.
5976 \end{doc}
5977
5978 \begin{doc}{
5979 cmd={name=mmzRegularPages, desc=readonly counter},
5980 }
5981 This counter holds the number of pages shipped out (so far) by the format's
5982 regular shipout routine. \emph{Do not change its value!}
5983
5984 In \hologo{LaTeX}, this counter is synonymous with
5985 \refcmd{ReadonlyShipoutCounter}, and in \hologo{ConTeXt}, it is synonymous
5986 with \refcmd{realpageno}. Memoize does not touch its value.
5987
5988 In \hologo{plainTeX}, Memoize hijacks the \refcmd{shipout} control sequence
5989 to count (and only to count) regular shipouts. In order for its value to be
5990 realistic, Memoize should be loaded before other packages which hack
5991 \cs{shipout} --- in particular, before \pkg{atbegshi}.
5992 \end{doc}
5993
5994 \begin{doc}{
5995 cmd={name=mmzExternPages, desc=readonly counter},
5996 }
5997 This counter holds the number of extern pages Memoize has shipped out (so
5998 far). \emph{Do not change its value!}
5999
6000 A third-party tool may inspect this counter to have a realistic count of
6001 shipped-out pages.
6002 \end{doc}
6003
6004 \begin{doc}{
6005 cmd={name=mmzExtraPages, desc=counter},
6006 }
6007 This counter holds the number of pages shipped out (so far) in a way not
6008 tracked by either \refcmd{mmzRegularPages} or \refcmd{mmzExternPages}. It
6009 \emph{should} be advanced by any code which performs such shipouts, or
6010 Memoize won't work correctly.
6011 \end{doc}
6012
6013
6014 \subsubsection{Tracing}
6015 \label{sec:ref:memoization:tracing}
6016
6017 \begin{doc}{
6018 key={name=trace, conditional=false},
6019 }
6020 When tracing is on, Memoize shows information about its decision processes on
6021 the terminal. You can learn whether the memoized code is being memoized,
6022 utilized or regularly compiled; find out the md5sum of the code and which
6023 input line it comes from; etc.
6024
6025 This key has the syntax of a conditional, but there is no underlying
6026 \hologo{TeX} conditional. The low-level interface for switching the tracing
6027 on and off consists of macros \docaux{cmd}{mmzTracingOn} and
6028 \docaux{cmd}{mmzTracingOff}.
6029
6030 To learn about tracing the ``auto'' part of the automemoization process, also
6031 \refcmd{AdviceTracingOn}.
6032 \end{doc}
6033
6034 \begin{doc}{
6035 key={name=include source in cmemo, conditional=true},
6036 }
6037
6038 As a courtesy towards a curious user and a debugging aid, Memoize can include
6039 a copy of the memo source in the c-memo. This feature is switched on by
6040 default, but as the package itself never uses that information, it can be
6041 safely switched off at any time.
6042 \end{doc}
6043
6044 \begin{doc}{
6045 key={name=include context in ccmemo, conditional=false},
6046 }
6047 When this key is in effect, the expanded context expression is appended to
6048 the cc-memo, behind the \docaux{cmd}{mmzThisContext} marker.
6049
6050 Memoize never uses the context information from the cc-memo; this information
6051 is only for tracing purposes.
6052 \end{doc}
6053
6054 \begin{doc}{
6055 key={name=direct ccmemo input, conditional=false},
6056 }
6057 When this key is set to false, a cc-memo is processed indirectly: it is first
6058 read into a token register, and it is the contents of this register which are
6059 executed. When the key is set to true, the cc-memo is simply |\input|ted.
6060
6061 The indirect execution is implemented to facilitate inverse search. Under
6062 the direct cc-memo input, inverse search pointed at an included extern will
6063 visit the cc-memo, which is not practical; under the indirect regime, the
6064 inverse search will work as expected, and this is why the indirect cc-memo
6065 input is the default.
6066
6067 The overhead produced by the default indirect input method seems negligible,
6068 but there are other factors which might make the user switch to the direct
6069 input. For one, a cc-memo changing some category codes will require direct
6070 input (no such cc-memos are ever produced out of the box). Less crucially,
6071 sometimes one would like to use the inverse search to figure whether a part
6072 of the document was produced by regular compilation or utilization, and which
6073 memos\slash externs were utilized if the latter. Figuring this out under the
6074 indirect input regime is harder: (i) reading the tracing information shown by
6075 \refmmz{trace} is the surest way to learn what's going on, although (ii)
6076 visual inspection of the externs and (iii) grepping through the |.memo.dir|
6077 folder for particular code often help, as well.
6078
6079 Both input methods use the same cc-memos; there is no need to
6080 \refmmz{recompile} the memos when switching the cc-memo input method. Note
6081 that the default indirect input method crucially relies on cc-memos ending
6082 with \refcmd{mmzEndMemo}; this macro should not appear in the cc-memo itself.
6083 \end{doc}
6084
6085
6086
6087 \subsubsection{Internal memo commands}
6088 \label{sec:ref:memoization:internal}
6089
6090 The end-user should never have to use these commands. They are not formally
6091 marked as internal by a |@| in their name only because doing so would
6092 complicate |\input|ting the memos due to the category code changes it would
6093 require.
6094
6095 \begin{doc}{
6096 cmd={name=mmzMemo},
6097 }
6098 This macro marks the beginning of a c-memo and a cc-memo core. Without it,
6099 utilization of a memo will not work.
6100 \end{doc}
6101
6102 \begin{doc}{cmd={name=mmzSource}}
6103 This macro marks the beginning of the memoized source in the c-memo. That
6104 source is not used by Memoize in any way. It's inclusion into the c-memo may
6105 be switched off by \refmmz{include source in cmemo}|=false|.
6106 \end{doc}
6107
6108 \begin{doc}{
6109 cmd={name=mmzResource, par=\marg{filename}},
6110 }
6111 This is an internal command, which only occurs in a cc-memo. It checks
6112 whether file \meta{filename} exists and is non-empty, and triggers
6113 recompilation of the memoized code if the check fails.
6114 \end{doc}
6115
6116 \begin{doc}{
6117 cmd={
6118 name=mmzIncludeExtern,
6119 par=\marg{seq}\braces{\refcmd{hbox}\alt\refcmd{vbox}}%
6120 \marg{expected width}\marg{expected height}\marg{expected depth}%
6121 \marg{padding left}\marg{padding bottom}\marg{padding right}\marg{padding top},
6122 }
6123 }
6124
6125 This is an internal command, which only occurs in a cc-memo. It includes the
6126 extern identified by the sequential number \meta{seq} into the document as a
6127 box of the specified type (horizontal or vertical). The extern is trimmed by
6128 the given padding values. After trimming, the command checks whether the
6129 size of the resulting box matches the given expectations; if it doesn't, a
6130 warning is yielded.
6131
6132 Before this command is executed, the externs should be listed by a sequence
6133 of \refcmd{mmzResource} commands; \meta{seq} refers to the sequential number
6134 of an extern in this sequence.
6135
6136 This command may also be executed by executing the entire contents of
6137 \refcmd{mmzCCMemo} after memoization.
6138 \end{doc}
6139
6140 \begin{doc}{
6141 cmd={name=mmzLabel, par=\marg{label key}\marg{label value}}
6142 }
6143 This is an internal command written into the cc-memo by the auto-handler of
6144 \refcmd{label}. It temporarily stores \meta{label value} into
6145 \refcmd{@currentlabel} and then executes \cs{label}\marg{label name}.
6146 \end{doc}
6147
6148 \begin{doc}{
6149 cmd={name=mmzEndMemo},
6150 }
6151 This macro marks the end of a cc-memo. It is used to grab the cc-memo core
6152 (everything between \refcmd{mmzMemo} and \refcmd{mmzEndMemo}) under the
6153 indirect cc-memo input regime, i.e.\ when \refmmz{direct ccmemo input} is not
6154 in effect.
6155 \end{doc}
6156
6157
6158 \subsection{Location of memos and externs}
6159 \label{sec:ref:dirs}
6160
6161 \begin{doc}{easy,
6162 key={name=memo dir,par=\meta{name}, desc={style, default \cs{jobname}}},
6163 }
6164 A convenient way to store memos and externs in a dedicated directory (see
6165 sections~\ref{sec:tut:memodir} and~\ref{sec:tut:multifile} for the tutorial
6166 and an illustration of the resulting directory structure). Note that since
6167 Memoize version 1.3.0, this key is initially in effect.
6168
6169 This key places memos and externs in subdirectory \meta{name}|.memo.dir| of
6170 the output directory, with \meta{name} defaulting to the document name.
6171 (Without an argument, these files are thus stored in subdirectory
6172 \meta{document name}|.memo.dir|.) Note that memo and extern filenames are
6173 set to contain no prefix; it is unnecessary, as \meta{name} already occurs in
6174 the directory name.
6175
6176 Under the hood, this style merely issues ||\refmmz{prefix}|={#1.memo.dir/}|
6177 (note the slash).
6178 \end{doc}
6179
6180 \begin{doc}{easy,key={name=no memo dir,par=\meta{name},
6181 desc={style, default=\cs{jobname}}}}
6182 A convenient way to undo the effect of \refmmz{memo dir}, such that memos and
6183 externs are placed in the output directory, with filenames prefixed by
6184 \meta{name}|.| (mind the dot), where \meta{name} defaults to the document
6185 name. See section~\ref{sec:tut:memodir} for an illustration of the resulting
6186 directory contents.
6187
6188 This style is an abbreviation for \code{\refmmz{prefix}={\meta{name}.}}.
6189 \end{doc}
6190
6191 \begin{doc}{
6192 key={name=prefix, par=\meta{prefix},
6193 desc={no default, initially \code{\cs{jobname}.memo.dir/}}},
6194 }
6195 This key determines the location of memo and extern files, and the initial
6196 part of their filenames.
6197
6198 If \meta{prefix} contains no |/|, memos and externs are stored in the output
6199 directory, alongside the output PDF, and their filenames begin with
6200 \meta{prefix} (and continue with the identifier, see below). For example,
6201 \refmmz{no memo dir} (without an argument) sets \refmmz{prefix} to
6202 \code{\cs{jobname}.}, i.e.\ the name of the document plus |.| (a dot). The
6203 resulting situation is depicted in the |dirty-house| example in
6204 section~\ref{sec:tut:memodir}.
6205
6206 If \meta{prefix} contains slashes, everything up to the final slash
6207 determines the directory which memos and externs will be stored into, and the
6208 part after the final slash determines the starting part of their filenames; a
6209 slash (|/|) as a directory separator should be used even on Windows, where
6210 the system directory separator is a backslash (|\|). For example,
6211 \refmmz{memo dir} (without an argument) sets \refmmz{prefix} to
6212 |\jobname.memo.dir/|. As shown in the |clean-house| example in
6213 section~\ref{sec:tut:memodir}, this results in memos and externs stored in
6214 directory \meta{jobname}|.memo.dir|, with their filenames consisting solely
6215 of the identifier.
6216
6217 In detail, the paths to memos and externs are constructed as shown below,
6218 where \meta{code md5 sum} and \meta{context md5 sum} identify the memoized
6219 code and the \refmmz{context} in effect at its memoization, and $N$ is the
6220 sequential number of the extern with respect to that code and context ($N$ is
6221 usually $0$, as memoization normally produces a single
6222 extern).\footnote{Normally, these paths are relative to the current directory
6223 (i.e.\ the directory where \hologo{TeX} is executed from; usually, this
6224 will be the directory where the compiled |.tex| file resides). However,
6225 when \hologo{TeX} is invoked with option |-output-directory|, these paths
6226 are relative to the specified output directory. Furthermore, when a memo
6227 or extern cannot be written into the current\slash output directory, it
6228 will be stored into the temporary directory \docref{TEXMFOUTPUT}, if
6229 specified.}
6230 \begin{center}
6231 \begin{tabular}{r@{ }l}
6232 c-memo:&\meta{prefix}\meta{code md5 sum}|.memo|\\
6233 cc-memo:&\meta{prefix}\meta{code md5 sum}|-|\meta{context md5 sum}|.memo|\\
6234 extern:&\meta{prefix}\meta{code md5 sum}|-|\meta{context md5 sum}-$N$|.pdf|\\
6235 &(where ``-$N$'' is only present when $N\neq 0$, i.e.\ for non-first externs)\\
6236 \end{tabular}
6237 \end{center}
6238
6239 \begin{tcolorbox}[warning,title=The final slash matters]
6240 To reiterate, the presence vs.\ absence of a slash (|/|) determines whether
6241 memos and externs are stored in a dedicated directory or not. For example,
6242 if you want to store memos and externs in subdirectory |memos| of the
6243 output directory, you should set \code{\refmmz{prefix}=memos/}, with the
6244 final slash. Without the final slash, these files would end up in the
6245 output directory.
6246 \end{tcolorbox}
6247
6248 In principle, the directory specified by this key must already exist.
6249 However, Memoize does it best to create it for you, and should succeed at
6250 least when extraction method \refmmz{extract=perl} or \refmmz{extract=python}
6251 can be used on the system. See also \refmmz{mkdir} and \refmmz{mkdir
6252 command}.
6253
6254 When invoked from the document body, each invocation of this key records the
6255 new prefix by invoking \refmmz[show keypath]{record/record type/prefix} for
6256 each active \meta{record type} (this typically results in a
6257 \refcmd{mmzPrefix} entry in the \dmmz file) and attempts to create memo
6258 directory if \refmmz{mkdir} is in effect. When \refmmz{prefix} is executed
6259 in the document preamble, these actions are only carried out at the beginning
6260 of the document, for the final value of the key.
6261
6262
6263 \end{doc}
6264
6265 \begin{doc}{key={name=mkdir, conditional=true}}
6266 When this key is set to true and the value of \refmmz{mkdir command} is
6267 non-empty, Memoize will attempt to create the memo directory set by
6268 \refmmz{prefix} if it does not yet exist.
6269
6270 The directory is created using the system command specified by \refmmz{mkdir
6271 command}. The directory creation takes place at the beginning of the
6272 document and at every subsequent invocation of key \refmmz{prefix}.
6273 \end{doc}
6274
6275 \begin{doc}{
6276 key={name=mkdir command, par=\meta{system command},
6277 desc={no default, initially \code{\refscript{memoize-extract.pl} \refscript{memoize-extract.pl--mkdir}}}},
6278 }
6279 This key sets the command used to create the memo directory specified
6280 by \refmmz{prefix} when \refmmz{mkdir} is in effect.
6281
6282 Memoize attempts to create the directory by executing \meta{system command}
6283 followed by (a space and) \meta{directory}, where \meta{directory} is the
6284 directory part of the \meta{prefix} set by \refmmz{prefix}. The directory
6285 creation takes place at the beginning of the document and at every subsequent
6286 invocation of key \refmmz{prefix}.
6287
6288 This key could be set to |mkdir| (on Linux, |mkdir -p| would be an even
6289 better choice), however, this is not advisable as |mkdir| does not respect
6290 \hologo{TeX}'s output restrictions, set by \docref{openout_any} in
6291 \TeXLive and |[Core]AllowUnsafeOutputFiles| in \MiKTeX.
6292 Further note that as the value of this key is a system command, an
6293 appropriate shell escape mode must be in effect to execute it successfully;
6294 again, not something to be taken lightly.
6295
6296 The extraction scripts shipped with Memoize accept option
6297 \refscript{memoize-extract.pl--mkdir}, which makes them behave as a safe
6298 variant of |mkdir|, i.e.\ a |mkdir| which respects \hologo{TeX}'s output
6299 restrictions. As most \hologo{TeX} setups will have access to at least a
6300 rudimentary Perl installation required to successfully execute
6301 \code{\refscript{memoize-extract.pl} \refscript{memoize-extract.pl--mkdir}},
6302 this command is the default value of \refmmz{mkdir command}. Therefore, in
6303 most cases, the memo directory (if set via \refmmz{prefix} or \refmmz{memo
6304 dir}) should be created under the initial settings without any user
6305 intervention.
6306
6307 Invoking \code{\refmmz{extract}=\refmmz{extract=python}} changes the value of
6308 \refmmz{mkdir command} to \code{\refscript{memoize-extract.py}
6309 \refscript{memoize-extract.pl--mkdir}}, but only if \refmmz{mkdir command}
6310 is currently set to the initial value, i.e.\ it will not effect the change if
6311 the user adjusted \refmmz{mkdir command} manually.
6312 \end{doc}
6313
6314
6315 \subsection{Extern extraction}
6316 \label{sec:ref:externs}
6317
6318 \begin{doc}{easy,
6319 key={name=extract,par=\marg{extraction method},
6320 desc={preamble-only, initially \refmmz{extract=perl}, default: see below}},
6321 }
6322 This key selects or executes the extern \meta{extraction method}, i.e. the
6323 method which Memoize will use to extract the extern pages out of the document
6324 PDF.
6325
6326 Out of the box, Memoize recognizes the following \meta{extraction method}
6327 keywords: \docAux{easy, of=key:/mmz/extract, value={name=perl}, comma,
6328 value={name=python}, comma, value={name=tex}, and, value={name=no}}. The
6329 first three keywords trigger extern extraction using the methods documented
6330 in sections~\ref{sec:ref:extraction-perl-python}
6331 and~\ref{sec:ref:extraction-tex}. The final keyword (not available in
6332 \hologo{plainTeX}) instructs Memoize to \emph{not} perform the extraction at
6333 the end of loading the package; it should be used when extraction is
6334 performed externally (for details, see section~\ref{sec:setup}). Additional
6335 methods may be installed by defining key \docaux{key
6336 path}{mmz/extract}|/|\meta{extraction method}.
6337
6338 When invoked from \reffile{memoize.cfg} or used as a package option, this key
6339 \emph{selects} the extraction method. In this case, the key has no default
6340 value, i.e.\ it is illegal to use it without an argument. The method
6341 selected by the package option overrides the method selected in
6342 \reffile{memoize.cfg}, which in turn overrides the package-initial value
6343 \refmmz{extract=perl}.
6344
6345 In \hologo{LaTeX} and \hologo{ConTeXt}, the selected method is automatically
6346 executed at the end of loading the package. This does not happen in
6347 \hologo{plainTeX}, because we want to allow the author to override the
6348 initial extraction method (\refmmz{extract=perl}) or the extraction method
6349 specified in \reffile{memoize.cfg}, even if \hologo{plainTeX} has no package
6350 options. In \hologo{plainTeX}, internal extraction must be triggered by an
6351 explicit invocation of \refmmz{extract} in the ``document preamble'' ---
6352 i.e.\ between |\input memoize| and \refcmd{mmzset}\braces{\refmmz{begin
6353 document}}. If given an \meta{extraction method} argument, the key will
6354 execute the given extraction method; otherwise, it will execute either the
6355 initial method, \refmmz{extract=perl}, or the overwide specified in
6356 \reffile{memoize.cfg}.
6357
6358 In general, Memoize guards against triggering the extraction more than once.
6359 In formats other than \hologo{plainTeX}, invoking key \refmmz{extract} from
6360 the document preamble is thus only allowed when
6361 \refmmz{extract}|=|\refmmz{extract=no} was previously selected. In this
6362 case, \refmmz{extract} in the document preamble behaves as in
6363 \hologo{plainTeX}, i.e.\ it triggers the given extraction method; if no
6364 extraction method is given, Memoize executes either the initial extraction
6365 method \refmmz{extract=perl}, or the extraction method specified in
6366 \reffile{memoize.cfg}. The possibility of invoking key \refmmz{extract} from
6367 the document preamble makes it possible to bake Memoize into a user format
6368 (with \refmmz{extract}|=|\refmmz{extract=no}), and trigger the internal
6369 extraction manually.
6370
6371 Executing extraction method \refmmz{extract=python} has an additional effect
6372 of setting \refmmz{mkdir command} to \code{\refscript{memoize-extract.pl}
6373 \refscript{memoize-extract.pl--mkdir}}, but only if \refmmz{mkdir command}
6374 is currently set to the initial value, i.e.\ it will not effect the change if
6375 the user adjusted \refmmz{mkdir command} manually.
6376 \end{doc}
6377
6378 \subsubsection{Perl- and Python-based extraction}
6379 \label{sec:ref:extraction-perl-python}
6380
6381 Perl- and Python-based extraction is triggered by
6382 \refmmz{extract}|=|\refmmz{extract=perl} and
6383 \refmmz{extract}|=|\refmmz{extract=python}, respectively.
6384
6385 \begin{doc}{
6386 key={name=perl extraction command, par=\meta{system command},
6387 desc={no default, initially \refscript{memoize-extract.pl}}},
6388 key={name=perl extraction options, par=\meta{options},
6389 desc={no default, initially: \code{\refscript[short]{memoize-extract.pl--format}
6390 \meta{format} \string\jobname}}},
6391 key={name=python extraction command, par=\meta{system command}, desc={no
6392 default, initially \refscript{memoize-extract.py}}}, key={name=python
6393 extraction options, par=\meta{options}, desc={no default, initially:
6394 \code{\refscript[short]{memoize-extract.pl--format} \meta{format}
6395 \string\jobname}}}
6396 }
6397
6398 These keys determine the system calls used for invoking the extraction
6399 scripts \refscript{memoize-extract.pl} and \refscript{memoize-extract.py}.
6400 All the details below apply both to the Perl and the Python version.
6401
6402 Use |perl|\slash|python extraction command| to set the name of the extraction
6403 script. If necessary, include the full path to the script, or |perl|\slash
6404 |python| plus the path to the script. Whatever you set here must be allowed
6405 by the shell escape mode.
6406
6407 Use |perl|\slash|python extraction options| to set the options that the
6408 script will receive; consult the documentation of
6409 \refscript{memoize-extract.pl} for their meaning. The initial value asks the
6410 script to process \meta{document name}\dmmz, and informs it that it is
6411 executed from within a \hologo{TeX} compilation of a document in the given
6412 \meta{format} (|latex| for \hologo{LaTeX}, |plain| for \hologo{plainTeX},
6413 |context| for \hologo{ConTeXt}); the latter makes the script yield any
6414 warnings or errors in a form expected by the format.
6415
6416 During the execution of a system call, the values of these settings are fully
6417 expanded. Furthermore, these keys were initialized using \pkg{pgfkeys}
6418 handler |.initial|, so their values may be modified by handlers |.prefix|,
6419 |.append|, etc. The initial value of the extraction options contains a space
6420 on both sides, so that these handlers are easy to use. For example, write
6421 \code{\refmmz{perl extraction
6422 options}/.append=\refscript{memoize-extract.pl--quiet}} to ask
6423 for less output.
6424 \end{doc}
6425
6426 \begin{doc}{
6427 script={name=memoize-extract.pl, par=[\meta{options}] \meta{name}\texttt{.mmz}},
6428 script={name=memoize-extract.py, par=[\meta{options}] \meta{name}\texttt{.mmz}},
6429 }
6430 These scripts extract the new externs recorded in \meta{name}\dmmz from
6431 \meta{name}|.pdf|. Memoize invokes them when loaded with package option
6432 \refmmz{extract}|=|\refmmz{extract=perl} (the default) or
6433 \refmmz{extract}|=|\refmmz{extract=python}.
6434
6435 Argument \meta{name}\dmmz may be given either in full (e.g.\ |doc.mmz|), or
6436 merely as the stem (|doc|). In the latter case, \dmmz is appended to the
6437 given argument even if it already contains a suffix (e.g.\ |my.doc| will
6438 result in |my.doc.mmz|); the exception is suffix |.tex|, which is replaced by
6439 \dmmz.
6440
6441 The script inspects the given record file, \meta{name}\dmmz, for lines of
6442 form \refcmd{mmzNewExtern}\marg{extern filename}\marg{extern page
6443 number}\braces{\meta{expected width}\texttt{pt}}\braces{\meta{expected
6444 height}\texttt{pt}}. For each such line, page number \meta{extern page
6445 number} is extracted from \meta{name}|.pdf| into \meta{extern filename}.
6446 (The script also creates directories specified by \refcmd{mmzPrefix} lines.
6447 Other lines are ignored, and so are commented invocations of
6448 \refcmd{mmzNewExtern} and \refcmd{mmzPrefix}.)
6449
6450 The \meta{extern filename} may contain a (relative or absolute) path to the
6451 new extern file. The relative paths are relative to the location of the
6452 \dmmz file, even when the script is invoked from some other directory.
6453
6454 To guard against extracting a wrong page, the script checks whether the size
6455 of each extracted page matches the \meta{expected width} and \meta{expected
6456 height}.\footnote{\label{fn:tolerance}To avoid false positives, the match
6457 need not be exact, a difference up to |0.01pt| is tolerated. Some PDF
6458 tools, notably the |PDF::API2| library deployed by the Perl version of the
6459 script, round the dimensions of a PDF page, recorded in |/MediaBox|, to two
6460 digits.} If it does not, the script refuses to extract the page, yields a
6461 warning and even removes the extern file if it exist.
6462
6463 The extraction script's paranoia extends further. It will refuse to extract
6464 the page, yielding a warning, if a (c)c-memo associated to the extern does
6465 not exist. And it will respect the |openin| and |openout| settings of
6466 |texmf.cnf|; under the default configuration, it will therefore refuse
6467 (yielding an error) to write to any file whose absolute path does not occur
6468 under the current directory, the temporary directory set by
6469 \docref{TEXMFOUTPUT} (in \reffile{texmf.cnf} or as an environment variable),
6470 or the output directory (\docref{TEXMF_OUTPUT_DIRECTORY}). Furthermore, it
6471 will refuse to write into the root directory (except on Windows, where
6472 writing into a drive root might potentially make sense).
6473
6474 Exit codes: 0 = success, 1 = Perl/Python error, 2 = usage error, 10 =
6475 extraction warning, 11 = extraction error.
6476
6477 \yadocset{ of=script:memoize-extract.pl, override/.style={ index
6478 annotation={option of {%
6479 \hypercolor{link}{gray}%
6480 \hyperref[script:memoize-extract.pl]{\texttt{memoize-extract}}%
6481 }}, }, }
6482 \begin{doc}{
6483 option={name=pdf, short name=P, par=\meta{pdf}}
6484 }
6485 Extract the externs from the given \meta{pdf} instead of the default
6486 \meta{name}\texttt{.mmz}. Note that file \meta{pdf}, despite the different
6487 name, should be produced by the same compilation that produced
6488 \meta{name}\dmmz, otherwise wrong pages might be extracted.
6489 \end{doc}
6490 \begin{doc}{
6491 option={name=prune, short name=p}
6492 }
6493 After extraction, remove the extracted extern pages from the document PDF.
6494 \end{doc}
6495 \begin{doc}{
6496 option={name=keep, short name=k}
6497 }
6498 By default, the script comments out the \refcmd{mmzNewExtern} lines in the
6499 \dmmz file, to prevent multiple extractions. Specifying this option
6500 prevents this behaviour.
6501 \end{doc}
6502 \begin{doc}{
6503 option={name=format, short name=F,
6504 par=\code{latex}\Alt\code{plain}\Alt\code{context}}
6505 }
6506 When this option is given, the script assumes that it was called from
6507 within a \hologo{TeX} compilation of a document in the given format
6508 (|latex| for \hologo{LaTeX}, |plain| for \hologo{plainTeX}, |context| for
6509 \hologo{ConTeXt}).
6510
6511 For one, the script will prefix all its output by the script name, to be
6512 easily identifiable in the terminal output, but more importantly, it will
6513 create a log file (\code{\string\jobname.mmz.log}) which will receive any
6514 warnings and errors yielded by the script (in absence of this option, the
6515 warnings and errors are printed to the standard error). The log contains
6516 messages in a form recognized by the format (e.g.\ \cs{PackageError} for
6517 \hologo{LaTeX}), so Memoize can then \cs{input}s this log to reproduce the
6518 warnings and errors in the compilation.
6519
6520 There should be no need to use this option when executing the script form
6521 the command line.
6522 \end{doc}
6523 \begin{doc}{
6524 option={name=force, short name=f}
6525 }
6526 Force extern extraction even if the size check or (c)c-memo existence check
6527 fails. The failure will still yield a warning.
6528 \end{doc}
6529 \begin{doc}{
6530 option={name=quiet, short name=q}
6531 }
6532 Normally, the script prints what it is doing to the standard output; in
6533 particular, it prints out the page number and the filename of each extern
6534 it is extracting. This option disables this behaviour.
6535 \end{doc}
6536 \begin{doc}{
6537 option={name=library, short name=L,
6538 par=\texttt{PDF::API2\Alt PDF::Builder}, desc=Perl-only}
6539 }
6540 Use the specified library for PDF processing. If this option is not given,
6541 the script uses the first available library (in the order given above).
6542 \end{doc}
6543 \begin{doc}{
6544 option={name=mkdir, short name=m}
6545 }
6546 When given this option, the extraction script transforms into a paranoid
6547 |mkdir| (|-p|). Argument \meta{name}\dmmz is interpreted as a path to the
6548 directory to create; all other options are ignored.
6549
6550 The ancestors of the directory are created as needed. The script will
6551 refuse to create any directory whose absolute path does not occur under the
6552 current directory or a directory listed in \docref{TEXMFOUTPUT} (set in
6553 \reffile{texmf.cnf} or as an environment variable).
6554
6555 This option exists so that the author using the restricted shell mode does
6556 not have to list |mkdir| among the restricted shell mode commands (and it
6557 is also safer than a plain |mkdir|).
6558
6559 Initially, \refmmz{mkdir command} is set to
6560 \code{\refscript{memoize-extract.pl}
6561 \refscript{memoize-extract.pl--mkdir}}.
6562 \end{doc}
6563 \begin{doc}{
6564 option={name=version, short name=V}
6565 }
6566 Show Memoize version.
6567 \end{doc}
6568 \begin{doc}{
6569 option={name=help, short name=h}
6570 }
6571 Show help.
6572 \end{doc}
6573
6574 Functionally, the Perl (|.pl|) and the Python (|.py|) version of the script
6575 are almost equivalent. The minor differences, listed below, are mostly due
6576 to the underlying PDF-processing library:
6577 \hreftt{https://metacpan.org/pod/PDF::API2}{PDF::API2} in Perl, and
6578 \hreftt{https://pypi.org/project/pdfrw2/}{pdfrw2} in Python.
6579 \begin{itemize}
6580 \item The Python script is about twice as fast as the Perl script. However,
6581 both scripts are very fast compared to the \hologo{TeX}-based extraction.
6582 On my computer, extracting 160 externs out of a 400-page book takes 1.4s
6583 with Python, 3.7s with Perl, and 65s with \hologo{TeX}. But even when
6584 using \hologo{TeX}-based extraction, externalization using Memoize is
6585 extremely fast compared to the \TikZ's externalization library. Adding
6586 the regular compilation time of one minute to the above numbers, we arrive
6587 at the maximum externalization time of about two minutes, whereas my
6588 estimate for the production of all 160 externs using \TikZ's
6589 externalization would be an \emph{hour} or more.
6590 \item The Python script cannot extract externs out of PDFs created without
6591 stream compression, i.e.\ with \refcmd{pdfvariable}
6592 \docref{pdfvar:compresslevel} set to |0|.
6593 \item Occasionally, the Perl script crashes during extraction externs; see
6594 section~\ref{sec:troubleshooting} for details.
6595 \end{itemize}
6596 \end{doc}
6597
6598 \subsubsection{\texorpdfstring{\hologo{TeX}}{TeX}-based extraction}
6599 \label{sec:ref:extraction-tex}
6600
6601 \hologo{TeX}-based extraction is triggered by
6602 \refmmz{extract}|=|\refmmz{extract=tex}.
6603
6604 \begin{doc}{
6605 key={name=tex extraction command, par=\meta{system command},
6606 desc={no default, initially |pdftex|}},
6607 key={name=tex extraction options, par=\meta{options},
6608 desc={no default, initially: see below}},
6609 key={name=tex extraction script, par=\meta{\hologo{TeX} code},
6610 desc={no default, initially: see below}},
6611 }
6612 \yadocset{
6613 before/.style={label prefix=cmd:te},
6614 index annotation={defined at \hologo{TeX} extraction},
6615 }
6616
6617 Together, these keys determine the system call used for invoking the
6618 \hologo{TeX}-based extraction script, \refscript{memoize-extract-one.tex}.
6619 (They were initialized using \pkg{pgfkeys} handler |.initial|, so their
6620 values may be modified by handlers |.prefix|, |.append|, etc.)
6621
6622 Memoize uses the resulting system call string at the following occasions.
6623 First, it executes it, once for each new extern in the \dmmz file, during the
6624 internal extraction, i.e.\ when it is loaded with package option
6625 \refmmz{extract}|=|\refmmz{extract=tex}. Second, it uses it to construct
6626 record files of types \refmmz{record=sh}, \refmmz{record=bat} and
6627 \refmmz{record=makefile} when recording of these types is requested.
6628
6629 Key \refmmz{tex extraction command} sets the \hologo{TeX} binary used for
6630 \hologo{TeX}-based internal extraction. By default, the \hologo{pdfTeX}
6631 engine |pdftex| is used; other sensible values for this key are |luatex| and
6632 |xetex|, or the program name including the path. Note that shell escape mode
6633 must be configured appropriately, and that
6634 \refscript{memoize-extract-one.tex} must be compiled with the
6635 \hologo{plainTeX} format.
6636
6637 The value of key \refmmz{tex extraction options} is passed as options to the
6638 \hologo{plainTeX} binary. As shown in the initial value in the frame below,
6639 it may use the temporary macro \docaux{cmd}{externbasepath}, which expands to
6640 the path to the extern without the |.pdf| suffix. This macro is available
6641 during internal \hologo{TeX}-based extraction and during the execution of
6642 \refmmz[show keypath]{record/record type/new extern} key for and \meta{record
6643 type}.
6644
6645 \tcbinputexample{tex-extraction-options}[.excerpt]{%
6646 listing only, no .listing, no attachment, title={The initial value of
6647 \refmmz{tex extraction options}},
6648 }
6649
6650 Key \refmmz{tex extraction script} defines the command-line script executed
6651 to extract the extern. The default value, shown in the frame below, invokes
6652 \refscript{memoize-extract-one.tex} after setting its parameter macros.
6653 Temporary macros \docAux{cmd={name=pagenumber}, comma, cmd={name=expectedwidth},
6654 and, cmd={name={expectedheight}}} are defined at the same occasions as
6655 \refcmd{te:externbasepath} above, i.e.\ during internal \hologo{TeX}-based
6656 extraction and during the execution of a \refmmz{record/record type/new
6657 extern} key. The initial value requests the extern to be of the same version as
6658 the main document, if possible;\footnote{As far as I know, it is impossible
6659 to access the version of the PDF being produced in \hologo{XeTeX}, i.e.\
6660 there are no registers \docref{reg:pdfmajorversion} and
6661 \docref{reg:pdfminorversion}. To request production of a specific version of
6662 PDF, \hologo{XeTeX} must be invoked by with command-line option
6663 \code{-output-driver 'xdvipdfmx -V $N$'}.} note that Memoize defines
6664 \docref{reg:pdfmajorversion} and \docref{reg:pdfminorversion} in \hologo{LuaTeX}.
6665
6666 \tcbinputexample{tex-extraction-script}[.excerpt]{%
6667 listing only, no .listing, title=The initial value of \refmmz{tex extraction
6668 script}, no attachment}
6669
6670 As the value of \refmmz{tex extraction script} is fully expanded when used,
6671 the initial value shown above must prevent the expansion of much code.
6672 Furthermore, the initial value varies with the \hologo{TeX} format, as
6673 indicated by the |.dtx| guards in the definition of
6674 \refcmd{meo:warningtemplate}.
6675 \end{doc}
6676
6677 \begin{doc}{
6678 script={
6679 head prefix={%
6680 (\texttt{pdf}\alt\texttt{lua}\alt\texttt{xe})\texttt{tex -jobname}
6681 \meta{extern filename}
6682 \texttt{"}\meta{parameters}
6683 \cs{input}\texttt{\textbraceleft}
6684 },
6685 name=memoize-extract-one.tex,
6686 head infix=\texttt{\textbraceright"},
6687 },
6688 }
6689 Compiling \refscript{memoize-extract-one.tex} with \hologo{plainTeX}
6690 produces an extern file containing a single (extern) page extracted from the
6691 document PDF.
6692
6693 Memoize invokes this script, once for each new extern appearing in the \dmmz
6694 file, when loaded with package option
6695 \refmmz{extract}|=|\refmmz{extract=tex}.
6696
6697 The desired \meta{extern filename} is given as the value of option |-jobname|
6698 of the \hologo{TeX} binary. To set the extraction \meta{parameters}, define
6699 the following macros before |\input|ting the file:
6700 \yadocset{
6701 index annotation=option of \texttt{memoize-extract-one.tex},
6702 before/.style={head prefix=\cs{def}, label prefix=cmd:meo},
6703 }
6704 \begin{doc}{cmd={name=fromdocument, parameters=\marg{document pdf filename}}}
6705 Defining this macro sets the filename of the PDF which the externs will be
6706 extracted from. The filename is relative to the working directory.
6707 \end{doc}
6708 \begin{doc}{cmd={name=pagenumber, parameters=\marg{number}}}
6709 Defining this macro sets the number of the page to extract. The first page
6710 has number 1.
6711 \end{doc}
6712 \begin{doc}{
6713 cmd={name=expectedwidth, parameters=\marg{dimension}, desc=optional},
6714 cmd={name=expectedheight, parameters=\marg{dimension}, desc=optional},
6715 }
6716 Defining these macros sets the expected width and height of the extracted
6717 page.
6718
6719 To guard against extracting a wrong page, the dimensions of the extracted
6720 page are compared against the expected width and height. If the size check
6721 fails,\footnote{The match need not be exact, see
6722 footnote~\ref{fn:tolerance}.} the resulting extern PDF is empty (which
6723 counts as non-existent when Memoize checks for its presence when it
6724 attempts to utilize it), and a warning message (formatted via
6725 \refcmd{meo:warningtemplate}) is printed to the log file, if logging was
6726 requested via \refcmd{meo:logfile}.
6727
6728 If any of these macros is undefined, the size check will be skipped.
6729 \end{doc}
6730 \begin{doc}{cmd={name=logfile, parameters=\marg{filename}, desc=optional}}
6731 Defining this macro sets the name of the log file. If not defined, no log
6732 file will be produced.
6733
6734 The log file is intended to be used when the script is invoked from an
6735 outer \hologo{TeX} compilation. In particular, it is intended to be
6736 |\input| by that compilation to see whether the extraction was successful.
6737 Upon a failed size check, it will contain a warning (formatted by
6738 \refcmd{meo:warningtemplate}, if that macro is defined). The log file ends
6739 with |\endinput| to signal that extraction actually took place.
6740 \end{doc}
6741 \begin{doc}{cmd={name=warningtemplate, parameters=\marg{code}, desc=optional}}
6742 Defining this macro determines how to log the warning message in the case
6743 of a failed size check. The macro should expand to a \hologo{TeX}
6744 format-specific warning message code containing the warning text given in
6745 \docAux{cmd={name=warningtext, label prefix=cmd:meo, into
6746 index=false}}.\footnote{Macro \refcmd{meo:warningtemplate} is passed
6747 the warning text by a macro rather than a formal parameter to avoid
6748 category code problems with the parameter character when setting key
6749 \refmmz{tex extraction script}.}
6750
6751 While the script formats the warning message \emph{text} on its own (``I
6752 refuse to extract page \dots''), the warning message is not written into
6753 the log unadorned. The log file is intended to be \cs{input} by the outer
6754 \hologo{TeX} compilation, and the idea is that inputting it should yield a
6755 warning in that compilation (in the case of a failed size
6756 check). Therefore, the content of the log file must contain an invocation
6757 of the command used to produce warning messages in the \hologo{TeX} format
6758 used by the outer compilation.
6759
6760 For example, when this script is invoked from within a \hologo{LaTeX}
6761 compilation, it makes sense to define something like
6762 |\def\warningtemplate{\PackageWarning{memoize}{\warningtext}}|.
6763 \end{doc}
6764 \begin{doc}{cmd={name=force,
6765 parameters=\bracestt{\texttt{true}\Alt\texttt{false}}, desc=optional}}
6766 If this macro is defined to |true|, extern extraction will be carried out
6767 even if the size-check fails. The failure will still be logged.
6768 \end{doc}
6769 \begin{doc}{
6770 cmd={name=mmzpdfmajorversion, parameters=\marg{number}, desc=optional},
6771 cmd={name=mmzpdfminorversion, parameters=\marg{number}, desc=optional},
6772 }
6773 Defining (one or both of) these macros requests that the extern PDF be
6774 produced with the given major\slash minor PDF version, i.e.\ the extraction
6775 script will set registers \docref{reg:pdfmajorversion} and
6776 \docref{reg:pdfminorversion}.
6777 \end{doc}
6778
6779 After extracting the extern, the script will end the compilation, i.e.\
6780 intentionally, only one page documents can be produced.
6781 \end{doc}
6782
6783 \subsubsection{The clean-up scripts}
6784 \label{sec:cleanup-scripts}
6785
6786 \begin{doc}{
6787 script={name=memoize-clean.pl, par=[\meta{options}] [\meta{name}\texttt{.mmz} \dots]},
6788 script={name=memoize-clean.py, par=[\meta{options}] [\meta{name}\texttt{.mmz} \dots]},
6789 }
6790 This script removes memo and extern files whose filenames start with
6791 \meta{prefix}es mentioned in the given \dmmz files or by the
6792 \refscript{memoize-clean.pl--prefix} option. Unless option
6793 \refscript{memoize-clean.pl--all} is given, the script only deletes the
6794 \emph{stale} files, i.e.\ the files not mentioned in any of the given \dmmz
6795 files.
6796
6797 A \meta{prefix} of a memo or an extern is what was set by key
6798 \refmmz{prefix}, or more commonly, one of the shortcut keys \refmmz{memo dir}
6799 and \refmmz{no memo dir}; see section~\ref{sec:ref:dirs} for details on the
6800 form of a memo/extern filename.
6801
6802 In detail, the script scans the given \dmmz files for occurrences of
6803 \refcmd{mmzPrefix}, and adds their \meta{prefix} arguments to the list of
6804 prefixes given on the command line by option
6805 \refscript{memoize-clean.pl--prefix}; a \meta{prefix} occurring in some \dmmz
6806 file is interpreted relatively to the location of the \dmmz file. The script
6807 removes all files whose full pathname (relative to the current directory)
6808 matches pattern
6809 \meta{prefix}\meta{md5sum}(|-|\meta{md5sum})(|.memo|\alt(|-|$N$)|.pdf|\alt|.log|),%
6810 \footnote{The |.log| files are produced by the \hologo{TeX}-based extraction
6811 script.} except those which occur as the \meta{filename} argument to one
6812 of \refcmd{mmzUsedCMemo}, \refcmd{mmzUsedCCMemo}, \refcmd{mmzUsedExtern},
6813 \refcmd{mmzNewCMemo}, \refcmd{mmzNewCCMemo} and \refcmd{mmzNewExtern} in one
6814 of the \dmmz files.
6815
6816 The script is fairly paranoid. It refuses to delete anything if a \dmmz file
6817 is malformed in any way (but not if it doesn't exist or is completely empty,
6818 which facilitates its usage in clean-up scripts), or if it would remove a
6819 file not residing under the current directory. Before removing the files, it
6820 lists the files to be removed and asks for confirmation.
6821
6822 Functionally, the Perl (|.pl|) and the Python (|.py|) version are completely
6823 equivalent.
6824
6825 \yadocset{
6826 of=script:memoize-clean.pl,
6827 override/.style={
6828 index annotation={option of {%
6829 \hypercolor{link}{gray}%
6830 \hyperref[script:memoize-clean.pl]{\texttt{memoize-clean}}%
6831 }},
6832 },
6833 }
6834 \begin{doc}{
6835 option={name=prefix, short name=p, par=\meta{prefix}}
6836 }
6837 Add \meta{prefix} to the list of prefixes; the given prefix is
6838 relative to the current directory. This option may be given multiple
6839 times.
6840 \end{doc}
6841 \begin{doc}{
6842 option={name=all, short name=a}
6843 }
6844 When given this option, the script removes \emph{all} memos and externs
6845 belonging to the document, not just the stale ones, i.e.\ it effectively
6846 ignores the occurrences of \refcmd{mmzUsedCMemo} and friends in the \dmmz
6847 file.
6848 \end{doc}
6849 \begin{doc}{
6850 option={name=yes, short name=y}
6851 }
6852 When given this option, the script does not ask for confirmation before
6853 removing the files.
6854 \end{doc}
6855 \begin{doc}{
6856 option={name=quiet, short name=q}
6857 }
6858 Normally, the script prints what it is doing to the standard output; in
6859 particular, it prints out the filename of each file as it is deleting it.
6860 This option disables this behaviour.
6861 \end{doc}
6862 \begin{doc}{
6863 option={name=help, short name=h}
6864 }
6865 Show help.
6866 \end{doc}
6867 \begin{doc}{
6868 option={name=version, short name=V}
6869 }
6870 Show Memoize version.
6871 \end{doc}
6872 \end{doc}
6873
6874 \subsubsection{Record files}
6875 \label{sec:ref:record-files}
6876
6877 \begin{doc}{
6878 key={name=record, par=\marg{record type},
6879 desc={cumulative, initially \refmmz{record=mmz}, no default}},
6880 key={name=no record},
6881 }
6882 Memoize records which externs were produced and used in the compilation,
6883 producing a record file of every type found in the record-type list. These
6884 keys add \meta{record type} to the record-type list, or clear this list. See
6885 section~\ref{sec:record-files} for details.
6886
6887 Note that passing an undefined \meta{record type} to this key will not yield
6888 an error.
6889
6890 Out of the box, the following \meta{record type}s are recognized:
6891 \yadocset{of=key:/mmz/record}
6892 \begin{doc}{value={name=mmz}}
6893 This record type produces a \dmmz file recording new/used
6894 externs/c-memos/cc-memos and changes in the \refmmz{prefix} to these
6895 files; see section~\ref{sec:.mmz} for details.
6896
6897 The produced file is named \docAux{file={name=mmz,
6898 name prefix=\meta{jobname}.,
6899 ref prefix=.,
6900 index prefix=.,
6901 }}. This name cannot be changed.
6902
6903 The \dmmz file is a \hologo{TeX} file, but uses only a simple subset of
6904 the \hologo{TeX} syntax, to be easily parsable by the external scripts such
6905 as \refscript{memoize-extract.pl}. Each line of the file consists of a
6906 (possibly commented) invocation of one of the commands listed below; the
6907 final line is |\endinput|. The \meta{prefix} below consists of the
6908 path to memos/externs and the immutable \refmmz{prefix} of their filename.
6909 \begin{doc}{
6910 cmd={name=mmzUsedCMemo, par=\marg{filename}},
6911 cmd={name=mmzUsedCCMemo, par=\marg{filename}},
6912 cmd={name=mmzUsedExtern, par=\marg{filename}},
6913 }
6914 Record that the (c)c-memo or extern residing in file \meta{filename} was
6915 utilized.
6916 \end{doc}
6917 \begin{doc}{
6918 cmd={name=mmzNewCMemo, par=\marg{filename}},
6919 cmd={name=mmzNewCCMemo, par=\marg{filename}},
6920 }
6921 Record that a new (c)c-memo residing in file \meta{filename} was
6922 produced.
6923 \end{doc}
6924 \begin{doc}{cmd={name=mmzNewExtern, par=\marg{filename}\marg{page number}%
6925 \marg{expected width}\marg{expected height}}}
6926 Record that a new extern was produced and dumped as page \meta{page
6927 number} into the document, that it should be extracted into file
6928 \meta{filename}, and that it should be \meta{expected width} wide and
6929 \meta{expected height} high (modulo tolerance of |0.01pt|, see
6930 footnote~\ref{fn:tolerance}), where the height is the total height
6931 comprising both \hologo{TeX} height and depth.
6932 \end{doc}
6933 \begin{doc}{cmd={name=mmzPrefix, par=\marg{prefix}}}
6934 Record that the \refmmz{prefix} of memo and extern files was
6935 changed.
6936 \end{doc}
6937 \end{doc}
6938 \begin{doc}{value={name=makefile}}
6939 This record type produces a makefile which, when processed by the |make|
6940 utility, triggers \hologo{TeX}-based extraction of the new externs.
6941 \begin{doc}{
6942 key={name=makefile, par=\marg{filename}, desc={no default,
6943 initially \texttt{memoize-extract.\string\jobname.makefile}}}
6944 }
6945 Use this key to change the filename of the produced makefile.
6946 \end{doc}
6947 \end{doc}
6948 \begin{doc}{value={name=sh}, value={name=bat}}
6949 These record types produce a shell script which, when executed, triggers
6950 \hologo{TeX}-based extraction of the new externs.
6951
6952 Use \refmmz{record=sh} on Unix-like systems, and \refmmz{record=bat} on
6953 Windows.
6954
6955 \begin{doc}{
6956 key={name=sh, par=\marg{filename}, desc={no default,
6957 initially \texttt{memoize-extract.\string\jobname.sh}}},
6958 key={name=bat, par=\marg{filename}, desc={no default,
6959 initially \texttt{memoize-extract.\string\jobname.bat}}}
6960 }
6961 Use these keys to change the filename of the produced shell script.
6962 \end{doc}
6963 \end{doc}
6964 \end{doc}
6965
6966 \begin{doc}{
6967 desc=definable,
6968 keypath=/mmz/record/\meta{record type},
6969 keypath label=/mmz/record/record type,
6970 name prefix={\meta{record type}/}, ref prefix=,
6971 index annotation={key in \texttt{/mmz/\meta{record type}}},
6972 key={name=begin},
6973 key={name=prefix, par=\marg{prefix}},
6974 key={name=new extern, par=\marg{filename}},
6975 key={name=new cmemo, par=\marg{filename}},
6976 key={name=new ccmemo, par=\marg{filename}},
6977 key={name=used extern, par=\marg{filename}},
6978 key={name=used cmemo, par=\marg{filename}},
6979 key={name=used ccmemo, par=\marg{filename}},
6980 key={name=end},
6981 }
6982 A new record type can be implemented by defining these keys in keypath
6983 \docAux{key path={name=mmz/record/\meta{record type}, label=mmz/record/record
6984 type}} (using the standard \pkg{pgfkeys} handlers such as |.code| and
6985 |.style|). The keys are invoked by Memoize where appropriate if recording
6986 for the defined type is activated by \refmmz{record}|=|\meta{record type},
6987 just as for the predefined types. Only those keys which are required for
6988 implementing the desired functionality need to be defined.
6989
6990 The following macros are available during the execution of key
6991 \refmmz{record/record type/new extern}:
6992 \yadocset{before/.style={label prefix=cmd:ne},
6993 index annotation=defined at \refmmz{record/record type/new extern}}
6994 \begin{doc}{cmd={name=pagenumber}}
6995 This macro holds the number of the extern page.
6996 \end{doc}
6997 \begin{doc}{
6998 cmd={name=expectedwidth},
6999 cmd={name=expectedheight},
7000 }
7001 These macros hold the width and the height of the extern page.
7002 \end{doc}
7003 \begin{doc}{cmd={name=externbasepath}}
7004 This macro holds the filename of the extern, minus the |.pdf| suffix (but
7005 including the path leading to the extern).
7006 \end{doc}
7007 \end{doc}
7008
7009
7010
7011 \subsection{Automemoization}
7012 \label{sec:ref:advicememoization}
7013
7014
7015 \subsubsection{Package Advice}
7016 \label{sec:ref:advice}
7017
7018 Package Advice is a namesake of \Emacs's Advice. As such, it implements a
7019 generic framework for extending the functionality of selected commands and
7020 environments. Each \emph{advised} command and environment is assigned a piece
7021 of \emph{advice} --- a command which is executed instead of the advised command
7022 and environment, and which may, or may not, invoke the original command or
7023 environment during its execution. The package offers an elegant way of
7024 declaring advice, setting up the conditions upon which the advised command will
7025 actually be replaced by the advice, collecting the arguments of the advised
7026 command and invoking it, and (de)activating the advice.
7027
7028 Before the advising framework can be used, it must be installed into a selected
7029 \pkg{pgfkeys} keypath (multiple installations into different keypaths are
7030 allowed, even if they handle the same commands). Memoize installs the
7031 framework into keypath \refkeypath{/mmz} and (primarily) uses it to
7032 automatically memoize the results of compilation of selected commands and
7033 environments.
7034
7035 \begin{doc}{
7036 keypath={/handlers}, desc=handler,
7037 key={name=.install advice,
7038 sort index=install advice,
7039 par=\marg{configuration keylist},
7040 },
7041 }
7042 This key is a \pkg{pgfkeys} key handler (see \PGFmanual{87.3.5}) which
7043 installs the advising framework into the keypath which it was invoked from
7044 --- henceforth, the \meta{namespace}.
7045
7046 For example, \code{\cs{pgfkeys}\bracestt{/my/\refkey{/handlers/.install
7047 advice}}} installs the framework into keypath |/my|.
7048
7049 Argument \meta{configuration keylist} may contain the following keys:
7050 \yadocset{keypath=/advice/install}
7051 \begin{doc}{key={
7052 name=setup key,
7053 parameters=\marg{name},
7054 description={no default, initially \texttt{advice}}},
7055 }
7056 This key determines the names of the user-interface keys used to setup
7057 advice for commands and environments in \meta{namespace}.
7058
7059 The keys whose names are determined by this key are the following:
7060 \meta{name}, \code{\meta{name} csname}, \code{\meta{name} key},
7061 \code{\meta{name}'}, \code{\meta{name} csname'} and \code{\meta{name}
7062 key'}. Memoize sets \refkey{/advice/install/setup key}|=auto|, and
7063 thereby defines \refmmz{auto}, \refmmz{auto csname}, \refmmz{auto key},
7064 \refmmz{auto'}, \refmmz{auto csname'} and \refmmz{auto key'}.
7065 \end{doc}
7066 \begin{doc}{key={
7067 name=activation,
7068 par=\meta{initial activation type},
7069 desc={no default, initially \texttt{immediate}}
7070 },
7071 }
7072 This key sets the \meta{initial activation type} for \meta{namespace}.
7073
7074 At the end of the installation, the system will execute
7075 \meta{namespace}|/|\refmmz{activation}|=|\meta{initial activation type};
7076 consequently, \meta{initial activation type} must be one of
7077 \refmmz{activation=immediate} and \refmmz{activation=deferred}. In
7078 Memoize, the \meta{initial activation type} is
7079 \refmmz{activation=deferred}.
7080
7081 Setting the \refmmz{activation} type during the installation only matters
7082 in \hologo{LaTeX}, where the installation ends by advising \refcmd{begin}
7083 to implement advising of environments.
7084 \end{doc}
7085
7086 \begin{tcolorbox}[warning]
7087 Writing the documentation for Advice, I was faced with a dilemma. Should
7088 the documentation reflect the fact that the full names of keys defined by
7089 the package depend on the installed instance of the framework, in
7090 particular on \meta{namespace} and \meta{setup key}? For example, should
7091 the reference headers contain things like \meta{namespace}|/activate| and
7092 \meta{namespace}|/|\meta{setup key} |csname|? In my opinion, this would
7093 make the reference hard to read, so I decided to have the reference headers
7094 refer to the Advice keys of the Memoize installation, where
7095 \meta{namespace}=|/mmz| and \meta{setup key}=|auto|, resulting in
7096 friendlier headers such as \refkeypath{/mmz}|/|\refmmz{activate} and
7097 \refkeypath{/mmz}|/|\refmmz{auto csname}. (Consequently, it also made
7098 sense to document Advice within the Memoize documentation.)
7099
7100 The bottomline: if you're reading this section with a non-Memoize
7101 installation in mind, you have to mentally replace any \refkeypath{/mmz}
7102 and \refmmz{auto} in the reference headers with the \meta{namespace} and
7103 the \meta{setup key} selected by that installation.
7104 (Section~\ref{sec:ref:advice:memoization} is another matter. Keys
7105 described there are only available in Memoize.)
7106 \end{tcolorbox}
7107
7108 In more detail, key handler \refkey{/handlers/.install advice} performs the
7109 following actions (as explained in the box above, we assume that the
7110 advising framework was installed into keypath \refkeypath{/mmz} with the
7111 setup key named |auto|):
7112 \begin{itemize}
7113 \item \edef\origtolerance{\the\tolerance}\tolerance=1000
7114 It defines the following keys in keypath \refkeypath{/mmz}:
7115 \refmmz{auto},
7116 \refmmz{auto csname},
7117 \refmmz{auto key},
7118 \refmmz{auto'},
7119 \refmmz{auto csname'},
7120 \refmmz{auto key'},
7121 \refmmz{activation},
7122 \refmmz{activate deferred},
7123 \refmmz{activate},
7124 \refmmz{deactivate},
7125 \refmmz{activate csname},
7126 \refmmz{deactivate csname}.
7127 \refmmz{activate key},
7128 \refmmz{deactivate key}.
7129 \refmmz{force activate},
7130 \refmmz{try activate}.
7131 \item
7132 It defines the following keys in keypath \refkeypath{/mmz/auto}:
7133 \refmmzauto{run conditions},
7134 \refmmzauto{outer handler},
7135 \refmmzauto{bailout handler},
7136 \refmmzauto{collector},
7137 \refmmzauto{args},
7138 \refmmzauto{collector options},
7139 \refmmzauto{clear collector options},
7140 \refmmzauto{raw collector options},
7141 \refmmzauto{clear raw collector options},
7142 \refmmzauto{inner handler},
7143 \refmmzauto{options},
7144 \refmmzauto{clear options},
7145 \refmmzauto{reset}.
7146 \item \tolerance\origtolerance\relax
7147 It defines the |.unknown| key handler for \refkeypath{/mmz/auto}.
7148 This handler appends any unknown keys (and their values) to
7149 \refmmzauto{options}.
7150 \item It executes \refmmz[show keypath]{activation}|=|\meta{initial
7151 activation type}.
7152 \item In \hologo{LaTeX}, it submits \refcmd{begin} to advising, thereby
7153 enabling environment support in this format. Consequently, advising of
7154 environments can switched off by writing \refmmz{deactivate}|=\begin|.
7155 \end{itemize}
7156 \end{doc}
7157
7158 \paragraph*{The keys installed into keypath \metabf{namespace}} are used to
7159 declare and (de)activate advice. In the documentation in this subsection,
7160 we assume that \meta{namespace}=\refkeypath{/mmz} and that \meta{setup
7161 key}=\refmmz{auto}. In particular, this also applies to the reference
7162 headers.
7163
7164 \begin{doc}{
7165 key={name=activation, par=\docAux{of=key:/mmz/activation, value={name=immediate},
7166 text=\Alt, value={name=deferred}}, desc={no default, initially set by \refkey{/handlers/.install advice}}},
7167 key={name=activate deferred, desc={style}},
7168 }
7169 Key \refmmz{activation} selects the activation regime. Under the
7170 \refmmz{activation=immediate} regime, keys \refmmz{activate},
7171 \refmmz{deactivate}, \refmmz{force activate} and \refmmz{try activate} behave
7172 as described in their documentation below. Under the
7173 \refmmz{activation=deferred} regime, however, those keys are not executed;
7174 rather, their invocations are appended to style \refmmz{activate deferred}.
7175 For example, writing \refmmz{activate}|=\foo| in the deferred activation
7176 regime appends \refmmz{activate}|=\foo| to \refmmz{activate deferred}. It is
7177 up to the user if and when to execute the keys collected in \refmmz{activate
7178 deferred}; see the documentation of \refmmz{manual} to learn what Memoize
7179 does with the contents of this style.
7180 \end{doc}
7181
7182
7183 \begin{doc}{easy,
7184 par=\marg{list of commands and/or environments},
7185 desc={style},
7186 key={name=activate},
7187 key={name=deactivate},
7188 }
7189 These keys activate or deactivate the advice for the given commands and
7190 environments. When the advice is activated, it replaces the advised command;
7191 when it is deactivated, the command is reverted to its original definition.
7192
7193 In Memoize, these keys are most commonly used to activate or deactivate
7194 automemoization for the given commands or environments. For example, write
7195 \refmmz{deactivate}|={\tikz,tikzpicture}| to deactivate automemoization of
7196 \TikZ pictures (which is declared and active by default). The curly braces
7197 may be omitted if the list contains a single command or environment, e.g.\
7198 \refmmz{deactivate}|=\tikz| or \refmmz{deactivate}|=tikzpicture|.
7199
7200 (De)activation of a piece of advice is completely orthogonal to its declaration with
7201 \refmmz{auto}. For example, there is no need to deactivate a command before
7202 redeclaring its advice, and reactivate it afterwards. A command may be
7203 activated even before declaring its advice --- however, the command itself
7204 must be defined at the time of activation.
7205
7206 As the advice is normally automatically activated upon declaration with
7207 \refmmz{auto}, explicit activation is rarely needed, but see \refmmz{auto'}.
7208 The effect of these keys under the deferred activation regime is described in
7209 \refmmz{activation}.
7210
7211 Note that I sometimes speak of (de)activating a command, and sometimes of
7212 (de)activating its advice. I mean the same thing.
7213 \end{doc}
7214
7215 \begin{doc}{
7216 par=\marg{control sequence name},
7217 desc={style},
7218 key={name=activate csname},
7219 key={name=deactivate csname},
7220 }
7221 These keys activate and deactivate a command given by its \meta{control
7222 sequence name}; for example, \refmmz{activate csname}|=foo| is equivalent
7223 to \refmmz{activate}|=\foo|. Note that unlike the regular \refmmz{activate}
7224 and \refmmz{deactivate}, their |csname| variants only accept a single command
7225 at a time (otherwise, including a comma in the command name would be
7226 impossible).
7227 \end{doc}
7228
7229 \begin{doc}{
7230 par=\marg{list of full key names},
7231 desc={style},
7232 key={name=activate key},
7233 key={name=deactivate key},
7234 }
7235 These keys activate and deactivate |pgfkeys| keys. Note that \emph{full} key
7236 names must be given, i.e.\ the names must include the keypath.
7237
7238 Under the hood, these keys merely execute \refmmz{activate} and
7239 \refmmz{deactivate} on the internal macros corresponding to the given keys.
7240 \end{doc}
7241
7242 \begin{doc}{
7243 key={name=try activate, conditional=false},
7244 }
7245 When this conditional is set to true, \refmmz{activate} will not yield an
7246 error if the advice is already activated, and \refmmz{deactivate} will not
7247 yield an error if the advice is not yet activated.
7248
7249 This key applies to the next, and only to the next, invocation of key
7250 \refmmz{activate} or \refmmz{deactivate}, i.e.\ it is reset back to |false|
7251 after invoking \refmmz{activate} or \refmmz{deactivate}.
7252 \end{doc}
7253
7254 \begin{doc}{
7255 key={name=force activate, conditional=false},
7256 }
7257 When this conditional is set to true, \refmmz{activate} will activate even a
7258 previously activated command, provided that, additionally, the command has
7259 been redefined since the prior activation.
7260
7261 In more detail, the original definition of the advised command is saved upon
7262 activation (to provide the possibility of both deactivation and the usage of
7263 the original command by the handler). Consequently, activation of an already
7264 activated command would result in the saved original definition being
7265 overwritten by the redefinition made during the first activation. However,
7266 if the handled command was meanwhile redefined by a third party, reactivation
7267 makes sense, under the assumption that the former original definition is
7268 obsolete and should be replaced by the (third party) redefinition. As a
7269 safeguard, however, \refmmz{activate} requires such reactivation to be
7270 explicitly requested using conditional \refmmz{force activate}.\footnote{A
7271 potential problem, not (yet) addressed, is that the third party might be
7272 another incarnation of the advising framework. In this case, forced
7273 reactivation will result in the loss of the original command and a circular
7274 dependency between the two pieces of advice.}
7275
7276 This key applies to the next, and only to the next, invocation of key
7277 \refmmz{activate}, i.e.\ it is reset back to |false| after invoking
7278 \refmmz{activate}. This key does not apply to \refmmz{deactivate}.
7279 \end{doc}
7280
7281 \begin{doc}{easy,
7282 key={name=auto, par=\marg{command or environment}{\marg{keylist}}, desc=style}}
7283
7284 This key sets up the advice for the given command or environment, or updates
7285 the configuration of an existing piece of advice.
7286
7287 In Memoize, this key is most commonly used to submit a command or environment
7288 to automemoization. For an environment (say, |bar|), it suffices to write
7289 \refmmz{auto}|=|\bracestt{bar}\bracestt{\refmmzauto{memoize}}; for a command,
7290 we usually need to include its argument specification:
7291 \refmmz{auto}|=\foo|\bracestt{\refmmzauto{memoize},
7292 \refmmzauto{args}=\bracestt{...}}. Another common usage is to prevent
7293 memoization during the execution of a command or environment:
7294 \refmmz{auto}|=|\bracestt{bar}\bracestt{\refmmzauto{nomemoize}}. For
7295 details, see sections~\ref{sec:tut:automemoization},
7296 \ref{sec:tut:working-on-a-picture} and~\ref{sec:tut:verbatim}.
7297
7298 Note that in \hologo{LaTeX}, advising an environment (say, |bar|) is
7299 different than advising the internal command (|\bar|) containing the
7300 \meta{begin-code} of the environment, created by \cs{newenvironment}.
7301 Advising environment |bar| effectively replaces the entire
7302 |\begin{bar}...\end{bar}| construction, so that \hologo{LaTeX}'s |\begin|
7303 code is never executed. The advice of command |\bar|, on the other hand,
7304 is executed after |\begin| initializes the environment; in particular, it
7305 is executed within the group introduced by the environment.
7306
7307 The advice is configured by the given \meta{keylist}, which is executed with
7308 the default keypath set to \docaux{key path}{mmz/auto}. Any unknown keys in
7309 \meta{keylist} are passed on to key \refmmzauto{options}; for example, a
7310 plain \refmmz{verbatim} or \refmmz{padding}|=2in| have the same effect as
7311 \refmmzauto{options}|=|\refmmz{verbatim} or
7312 \code{\refmmzauto{options}=\braces{\refmmz{padding}=2in}}.
7313
7314 This key automatically activates the declared advice, unless it is already
7315 activated; under the deferred activation regime, the automatic activation is
7316 deferred as well. Use variant \refmmz{auto'} when you don't want to
7317 automatically activate the advice.
7318
7319 When this key is used on a command or environment with an existing piece of advice,
7320 the advice is merely updated. This makes it easy to, for example,
7321 temporarily switch to verbatim collection of an environment in Memoize:
7322 \refmmz{auto}|=|\bracestt{tcolorbox}\braces{\refmmz{verbatim}}. Use key
7323 \refmmzauto{reset} to setup the advice from scratch:
7324 \refmmz{auto}|=|\bracestt{...}\bracestt{\refmmzauto{reset}, ...}.
7325
7326 A piece of advice consists of several interlocked components, declared by keys
7327 residing in path \refkeypath{/mmz/auto}: \refmmzauto{run conditions},
7328 \refmmzauto{bailout handler}, \refmmzauto{outer handler},
7329 \refmmzauto{collector} and \refmmzauto{inner handler}. During the execution
7330 of the advice, these components are available through the following macros:
7331 \refcmd{AdviceRunConditions}, \refcmd{AdviceBailoutHandler},
7332 \refcmd{AdviceOuterHandler}, \refcmd{AdviceCollector} and
7333 \refcmd{AdviceInnerHandler}. These macros are also defined during setup, and
7334 it is possible to change the configuration by modifying them directly; in
7335 that case, it likely also makes sense to use the low-level variant of this
7336 key, macro \refcmd{AdviceSetup}.
7337
7338 Control sequences used in the advice components do not need to be defined at
7339 the time of invoking key \refmmz{auto}, or activating the advice; they must
7340 only be defined at the time the advice is actually executed. It is thus
7341 perfectly fine to declare \refmmzauto{inner handler}|=\myinnerhandler| before
7342 defining |\myinnerhandler|, or to redefine |\myinnerhandler| between the
7343 invocations of the advised command.
7344
7345 This key configures not only the components, but also the options of the
7346 advice. These options are set by keys \refmmzauto{args},
7347 \refmmzauto{collector options}, \refmmzauto{raw collector options} and
7348 \refmmzauto{options}. Whether these options are used or not depends on the
7349 advice components. (Same as the components, the options have their
7350 corresponding low-level macros: \refcmd{AdviceArgs},
7351 \refcmd{AdviceCollectorOptions}, \refcmd{AdviceRawCollectorOptions} and
7352 \refcmd{AdviceOptions}.)
7353
7354 Parameter symbols (i.e.\ |#|) are not allowed in advice settings.
7355
7356 A command or environment may be submitted to several instances of the
7357 advising framework, i.e.\ instances installed under different keypaths. The
7358 effect of such chained advice depends on the order of activation. If advice
7359 $A$ is activated before advice $B$, it will also be applied before $B$.
7360
7361 The advice setup takes place in a group. Use key \refmmzauto{after setup} to
7362 execute code outside this group.
7363
7364 In general, the name of this key equals whatever was submitted to
7365 \refkey{/advice/install/setup key} during the installation of the advising
7366 framework via \refkey{/handlers/.install advice}; the initial value is
7367 |advice| (and Memoize sets the value to |auto|).
7368 \end{doc}
7369
7370 \begin{doc}{
7371 key={name=auto csname, par=\marg{control sequence name}{\marg{keylist}}, desc=style}
7372 }
7373 This key is a variant of \refmmz{auto}, but with the command of the first
7374 argument given as a control sequence name, i.e.\ \refmmz{auto
7375 csname}|={foo}{...}| is equivalent to \refmmz{auto}|=\foo{...}|.
7376 \end{doc}
7377
7378 \begin{doc}{
7379 key={name=auto key, par=\marg{full key}{\marg{keylist}}, desc=style},
7380 }
7381 This key is a variant of \refmmz{auto}, but it works with \pkg{pgfkeys} keys.
7382 The first argument should be a \meta{full key} like |/tcb/float|, i.e.\ it
7383 must consist of both the keypath and the keyname.
7384
7385 This key sets up advice for the internal command corresponding to the given
7386 \meta{full key}, and also properly initializes the collector, so that
7387 \refmmzauto{inner handler} will ``just work.''
7388 \end{doc}
7389
7390 \begin{doc}{
7391 key={name=auto', par=\marg{command or environment}{\marg{keylist}}, desc=style},
7392 key={name=auto csname', par=\marg{control sequence name}{\marg{keylist}}, desc=style},
7393 key={name=auto key', par=\marg{full key}{\marg{keylist}}, desc=style},
7394 }
7395 These keys are variants of \refmmz{auto}, \refmmz{auto csname} and
7396 \refmmz{auto key} which do not attempt to activate the command after setting
7397 it up.
7398 \end{doc}
7399
7400 \begin{doc}{
7401 cmd={
7402 name=AdviceSetup,
7403 par=\marg{namespace}\marg{command or environment}{\marg{setup code}}
7404 }}
7405 This macro is the low-level variant of key \refmmz{auto}. The differences
7406 between the two are the following:
7407 \begin{itemize}
7408 \item An invocation of the macro must provide the namespace (i.e.\ the
7409 installation keypath) as the first argument.
7410 \item There is no automatic activation at the end of the setup.
7411 \item The final argument should not be a keylist (of keys belonging to
7412 \refkeypath{/mmz/auto}) but \hologo{TeX} code adjusting the contents of the
7413 settings macros \refcmd{AdviceRunConditions}, \refcmd{AdviceBailoutHandler},
7414 \refcmd{AdviceOuterHandler}, etc. For the full list of available macros, see
7415 the documentation of their corresponding keys below; the setting macros are
7416 mentioned at the end of each entry.
7417 \end{itemize}
7418 \end{doc}
7419
7420 \begin{doc}{
7421 cmd={name=AdviceTracingOn},
7422 cmd={name=AdviceTracingOff},
7423 }
7424 Advice tracing is initially off. When it is on, Advice will show (on the
7425 terminal and in the |.log| file) which advice components are executed, and
7426 what arguments and options they have received.
7427 \end{doc}
7428
7429 \paragraph*{The keys installed into keypath \metabf{namespace}\texttt{/}\metabf{setup key}}
7430 are used to configure advice. They may only occur within the second
7431 argument of the setup key. In the documentation in this subsection, we assume
7432 that \meta{namespace}=\refkeypath{/mmz} and that \meta{setup
7433 key}=\refmmz{auto}. In particular, this also applies to the reference
7434 headers.
7435
7436 \begingroup
7437 \yadocset{keypath=/mmz/auto}
7438
7439 \begin{doc}{%
7440 key={name=run conditions, par=\meta{\hologo{TeX} code},
7441 desc=initially and default: \refcmd{AdviceRuntrue}},
7442 }
7443
7444 This key declares the \meta{control sequence} as the run conditions component
7445 of the advice.
7446
7447 The run conditions macro is executed at the very start of the advice. Its
7448 function is to decide whether we should proceed to advise the command by
7449 executing the outer handler, or execute the original command (after invoking
7450 the bailout handler).
7451
7452 The run conditions macro should take no arguments. If it determines that the
7453 run conditions are satisfied, it should set the \hologo{TeX} conditional
7454 \docaux{cmd}{ifAdviceRun} to true by executing \docaux{cmd}{AdviceRuntrue}.
7455 There is no need to execute \docaux{cmd}{AdviceRunfalse} when the run
7456 conditions are not satisfied.
7457
7458 Initially, the run conditions are set to \refcmd{AdviceRuntrue}, translating to
7459 ``always run.'' For two non-trivial examples, see \refmmzauto{run if
7460 memoization is possible} and \refmmzauto{run if memoizing}. Executing this
7461 key without a value restores it to the initial value.
7462
7463 During advising and advice setup, the run conditions of the advised command
7464 are accessible through \docaux{cmd}{AdviceRunConditions}, a parameterless
7465 macro expanding to the given \meta{\hologo{TeX} code}.
7466 \end{doc}
7467
7468 \begin{doc}{%
7469 key={name=bailout handler, par=\meta{\hologo{TeX} code},
7470 desc=initially and default: \cs{relax}},
7471 }
7472
7473 This key declares the \meta{\hologo{TeX} code} as the bailout handler
7474 component of the advice.
7475
7476 The bailout handler is executed when the run conditions are not met, just prior
7477 to executing the original definition of the advised command. The bailout
7478 handler should take no arguments.
7479
7480 The initial bailout handler, |\relax|, does nothing. Memoize defines and
7481 uses a bailout handler which clears the next-options. Executing this key
7482 without a value restores it to the initial value.
7483
7484 During advising and advice setup, the bailout handler of the handled command is
7485 accessible through \docaux{cmd}{AdviceBailoutHandler}, a parameterless macro
7486 expanding to the given \meta{\hologo{TeX} code}.
7487 \end{doc}
7488
7489 \begin{doc}{
7490 key={name=outer handler,par=\meta{\hologo{TeX} code},
7491 desc=initially and default: \refcmd{AdviceCollector}},
7492 }
7493
7494 This key declares the \meta{\hologo{TeX} code} as the outer handler component
7495 of the advice.
7496
7497 The outer handler can be safely imagined as the command which replaces the
7498 handled command. This also holds for handled environments, but with a
7499 caveat: for a \hologo{plainTeX} or \hologo{ConTeXt} environment |foo|, the
7500 outer handler replaces |\foo| and |\startfoo|, respectively; in the case of a
7501 \hologo{LaTeX} environment, it replaces |\begin{foo}|.%]
7502
7503 The outer handler is the first component which has the opportunity to inspect
7504 the arguments given to the handled command. It is invoked just in front of
7505 these arguments (which are, in case \hologo{TeX} hasn't seen them yet,
7506 untokenized), and while it is expected that the advice will consume
7507 the same arguments as the advised command itself would, how precisely that
7508 happens may vary from situation to situation. In particular, the argument
7509 structure of the outer handler is not prescribed.
7510
7511 In fact, the outer handler has complete control over the remainder of the
7512 advising process. In situations where advising requires knowledge of the
7513 advised command's arguments as a whole, the outer handler executes the
7514 collector, which in turn invokes the inner handler, which does the real work;
7515 see \refmmzauto{memoize} for the usage case which inspired this design.
7516 Sometimes, however, it is the outer handler which does the real work (and
7517 there is thus no inner handler). This is the case in situations when the
7518 arguments of the handled command are irrelevant for the functioning of the
7519 advice, or when the advice needs to inspect some individual argument of the
7520 handled command; for examples of such situations, see \refmmzauto{abort} and
7521 \refmmzauto{ref}.
7522
7523 To reiterate the argument situation of the outer handler, it sees the
7524 arguments of the handled command as they were given. The arguments are
7525 \emph{not} collected before invoking the outer handler --- in fact, avoiding
7526 the argument collection is the raison d'être of the outer handler! (In the case
7527 of an advised environment, the environment body can be seen as an argument of
7528 \hologo{LaTeX3} type \docref{xparse:+}\docref{xparse:b}.)
7529
7530 The outer handler (and any other component of the advice it invokes) has
7531 access to the following auxiliary macros, defined by the framework:
7532 \begin{itemize}
7533 \item \tolerance 1000 the macros holding the configuration of the advised
7534 command, as set up by \refmmz{auto}: \refcmd{AdviceRunConditions},
7535 \refcmd{AdviceBailoutHandler} and \refcmd{AdviceOuterHandler} are probably
7536 useless, as they refer to components already invoked, but the remaining
7537 components (\refcmd{AdviceCollector} and \refcmd{AdviceInnerHandler}) and
7538 their options (\refcmd{AdviceArgs}, \refcmd{AdviceCollectorOptions},
7539 \refcmd{AdviceRawCollectorOptions} and \refcmd{AdviceOptions}) should be
7540 commonly used.
7541 \item the macros holding information about the namespace and the advised
7542 command or environment: \refcmd{AdviceNamespace}, \refcmd{AdviceName},
7543 \refcmd{AdviceCsname}, \refcmd{AdviceReplaced} and \refcmd{AdviceOriginal}.
7544 (Commands \refcmd{AdviceGetOriginal} and \refcmd{AdviceCsnameGetOriginal}
7545 might also be useful, although using \refcmd{AdviceOriginal} will likely be
7546 more practical.)
7547 \end{itemize}
7548
7549 This key is initially set to \refcmd{AdviceCollector}; in other words,
7550 the initial outer handler leaves all the work to the collector and the inner
7551 handler. There is no need to specifically set up the outer handler when
7552 using the inner handler. Executing this key without a value restores it to
7553 the initial value.
7554
7555 During advising and advice setup, the outer handler of the advised command is
7556 accessible through \docaux{cmd}{AdviceOuterHandler}, a parameterless macro
7557 expanding to the given \meta{\hologo{TeX} code}.
7558 \end{doc}
7559
7560 \begin{doc}{
7561 key={name=collector,par=\meta{\hologo{TeX} code},
7562 desc=initially and default: see below},
7563 }
7564
7565 This key declares the \meta{\hologo{TeX} code} as the collector component of
7566 the advice.
7567
7568 The collector, if used, is invoked by the outer handler. It is invoked
7569 immediately in front of the advised command's arguments (which are, in case
7570 \hologo{TeX} hasn't seen them yet, untokenized), and its function is to
7571 collect these arguments and pass them on, as a single argument, to the inner
7572 handler.
7573
7574 While this manual occasionally states that the initial argument collector is
7575 \refcmd{CollectArguments} of package CollArgs, this is, if we're precise,
7576 incorrect on two counts. For one, the initial collector is not a CollArgs
7577 command, but an internal macro which acts as the ``bridge'' between Advice
7578 and CollArgs. Second, the initial collector does not really invoke
7579 \refcmd{CollectArguments}, but its cousin, \refcmd{CollectArgumentsRaw},
7580 which allows Advice (and Memoize) to fine tune its behaviour
7581 using the fast low-level (``programmer interface'') commands rather than the
7582 slower \pkg{pgfkeys} interface; clearly, the latter point also provides
7583 raison d'être for \refmmzauto{raw collector options}. Summing up, this key
7584 is initially set to an internal control sequence which compiles the settings
7585 provided by \refmmzauto{args}, \refmmzauto{collector options} and
7586 \refmmzauto{raw collector options} into an invocation of
7587 \refcmd{CollectArgumentsRaw} of package CollArgs.\footnote{The initial
7588 collector also sets the CollArgs' option \refcollargs{caller} to the
7589 name of the advised command or environment.} Executing this key without a
7590 value restores it to the initial value.
7591
7592 The above-mentioned collector settings were clearly tailored to suit
7593 \refcmd{CollectArgumentsRaw}. In general, a collector might or might not use
7594 them, and if it does, it may interpret them in any way. For example, Advice
7595 ships with a \refcmd{tikz} collector, \docaux{cmd}{mmzCollectTikZArguments}, which
7596 ignores them completely, as it knows everything about the idiosyncrasies of
7597 that command anyway. Incidentally, \refcmd{mmzCollectTikZArguments} becomes
7598 available upon loading \reffile{advice-tikz.code.tex} (which Memoize does
7599 automatically in the presence of \TikZ).
7600
7601 The collector has access to the same auxiliary macros as the outer handler.
7602 In particular, it will \emph{have} to use \refcmd{AdviceInnerHandler} (followed
7603 by the braced collected arguments) to invoke the inner handler.
7604
7605 During advising and advice setup, the collector of the advised command is
7606 accessible through \docaux{cmd}{AdviceCollector}, a parameterless macro
7607 expanding to the given \meta{\hologo{TeX} code}.
7608 \end{doc}
7609
7610 \begin{doc}{easy,
7611 key={name=args,par=\meta{argument specification}, desc=initially and default: unset},
7612 }
7613
7614 This key describes the \meta{argument specification} of the advised command.
7615
7616 The value of this key is intended to be interpreted by the
7617 \refmmzauto{collector}. Assuming this key retained its initial value, the
7618 given \meta{argument specification} is eventually interpreted by command
7619 \refcmd{CollectArguments} of package CollArgs, which expects an argument
7620 specification in the format specified by \hologo{LaTeX3}, also allowing for
7621 the uncommon\slash historical types only defined upon loading package
7622 \pkg{xparse}, and several CollArgs-specific extensions. For further
7623 information, see the following sources:
7624 \begin{itemize}
7625 \item The \hologo{LaTeX3} version
7626 of \href{https://www.latex-project.org/help/documentation/usrguide.pdf}
7627 {``\LaTeX\ for authors''} documents the standard argument types.
7628 \item The \pkg{xparse} manual additionally documents the uncommon\slash
7629 historical types only available upon loading package \pkg{xparse}.
7630 \item CollArgs extensions are documented in section~\ref{sec:ref:collargs}.
7631 \item For convenience, the entire \hologo{LaTeX3} argument specification
7632 format, complete with \pkg{xparse} and CollArgs extensions, is summarized
7633 in the frame on page~\pageref{tab:argspec} in
7634 section~\ref{sec:ref:collargs}.
7635 \end{itemize}
7636
7637 If \meta{argument specification} is not provided, the initial collector
7638 assumes that the advised command was defined using \hologo{LaTeX3}'s
7639 \refcmd{NewDocumentCommand} (or similar), and will attempt to retrieve the
7640 argument specification automatically.\footnote{The automatic retrieval
7641 deploys command \refcmd{GetDocumentCommandArgSpec} of package \pkg{xparse},
7642 which is the sole reason that Advice loads this package in \hologo{LaTeX}.}
7643
7644 In general, however, an argument collector may this interpret this setting
7645 in any way it sees fit --- or not at all. For example, in Memoize the value
7646 of \refmmzauto{args} is ignored for command \refcmd{tikz}, which requires a special
7647 collector (\refcmd{mmzCollectTikZArguments}).
7648
7649 When setting up advice for a \emph{command}, this key is initially ``unset,''
7650 i.e.\ it holds a special value indicating that the argument specification is
7651 not provided. Note that this special value is not an empty string;
7652 \refmmzauto{args}|={}|, or simply \refmmzauto{args}|=|, indicates a command
7653 which takes no arguments. During the execution of the advice, one may use
7654 the \hologo{LaTeX}-style conditional \docaux{cmd}{AdviceIfArgs}\marg{true
7655 branch}\marg{false branch} to test whether the argument specification was
7656 provided. Executing this key without a value restores it to the initial,
7657 unset value.
7658
7659 When setting up the advice of an \emph{environment}, this key is
7660 initialized to \docref{xparse:+}\docref{xparse:b} (a long environment body),
7661 making it unnecessary to specify this value manually. Note that this holds
7662 even for environments with arguments other that the environment body
7663 ``argument'': those arguments will be caught as the start of the body even if
7664 not explicitly specified.
7665
7666 During advising and advice setup, the argument specification of the advised
7667 command is accessible through \docaux{cmd}{AdviceArgs}, a parameterless macro
7668 expanding to the given \meta{argument specification}.
7669 \end{doc}
7670
7671
7672 \begin{doc}{
7673 key={name=collector options, par=\marg{keylist},
7674 desc={cumulative, initially empty, value required}},
7675 key={name=raw collector options, par=\marg{code},
7676 desc={cumulative, initially empty, value required}},
7677 }
7678 These keys append the given value to the list of user-friendly and raw
7679 collector options, respectively. A comma is prefixed to the user-friendly
7680 \meta{keylist} before appending it.
7681
7682 Both kinds of collector options are intended to be used by the collector,
7683 which may interpret them in any way it sees fit --- or not at all. The
7684 initial collector, which invokes \refcmd{CollectArgumentsRaw} of package
7685 CollArgs, passes both lists to this command, which interprets
7686 \refmmzauto{collector options} as a user-friendly \pkg{pgfkeys} keylist
7687 (which therefore requires a bit of processing) and \refmmzauto{raw collector
7688 options} as plain \hologo{TeX} code (expecting it to contain only the
7689 allowed, ``programmer's interface'' macros).\footnote{Clearly,
7690 \refmmzauto{raw collector options} are why Advice deploys
7691 \refcmd{CollectArgumentsRaw} rather than \refcmd{CollectArguments}. But
7692 how does it then pass the user-friendly \refmmzauto{collector options} to
7693 that command? It embeds them in \refcmd{collargsSet}.} The raw variant is
7694 used internally by both Advice and Memoize, and may be used by a package
7695 deploying the advising framework which wants to save a few processing cycles.
7696 In CollArgs, the two kinds of options are functionally equivalent; both are
7697 documented in section~\ref{sec:ref:collargs}.
7698
7699 Initially, the list of collector options is empty, and for commands, so is
7700 the list of raw collector options. For environments, however, the latter
7701 list is initialized to set (the raw equivalent of) \refcollargs{environment}
7702 to the environment name, and \refcollargs{end tag} to true. The rationale
7703 for the latter is that the environment body containing the end tag (e.g.\
7704 |\end{foo}|) is nicely compatible with \refcmd{AdviceReplaced} (which equals
7705 the begin tag, e.g.\ |\begin{foo}|) and \refcmd{AdviceOriginal} (which executes
7706 the original definition of e.g.\ |\begin{foo}|). For example, thanks to
7707 \refcollargs{end tag}, writing \refcmd{AdviceOriginal}|#1| in the inner
7708 handler executes the original environment. Importantly, the original
7709 environment can be executed without explicitly referring to the
7710 environment's name, and with code that works not only for environments of
7711 any \hologo{TeX} format, but is actually the same as the code which invokes
7712 an original \emph{command}. Consequently, the same inner handler works for
7713 both commands and environments, and in all \hologo{TeX} formats.
7714
7715 Furthermore, the initial collector also sets option \refcollargs{caller}
7716 to the name of the advised command or environment (however,
7717 \refcollargs{caller} never appears in any of the collector options
7718 lists; it is simply prefixed to them while constructing the invocation of
7719 \refcmd{CollectArgumentsRaw}). And in Memoize, using keys
7720 \refmmz{verbatim}, \refmmz{verb} or \refmmz{no verbatim} triggers the
7721 addition of the cognominal \refcollargs{verbatim}, \refcollargs{verb}
7722 or \refcollargs{no verbatim} among the collector options.
7723
7724 Precious few CollArgs' options thus remain to be set by the author. For
7725 memoization, the most likely candidates are \refcollargs{ignore nesting}
7726 and \refcollargs{ignore other tags}, which could help deal with unusual
7727 environments. Overriding the initial \refcollargs{end tag} by
7728 \refcollargs{begin tag}, \refcollargs{end tag} and/or \refcollargs{tags}
7729 might also be useful on occasion.
7730
7731 During advising and advice setup, the \pkg{pgfkeys} and the raw collector
7732 options of the advised command are accessible through
7733 \docaux{cmd}{AdviceCollectorOptions} and
7734 \docaux{cmd}{AdviceRawCollectorOptions}, both a parameterless macro expanding
7735 to the given \meta{keylist} and \meta{code}, respectively.
7736 \end{doc}
7737
7738 \begin{doc}{
7739 key={name=clear collector options},
7740 key={name=clear raw collector options},
7741 }
7742 These keys empty the list of user-friendly and raw collector options,
7743 respectively.
7744 \end{doc}
7745
7746 \begin{doc}{
7747 key={name=inner handler, par=\meta{\hologo{TeX} code}, long description=0.5,
7748 desc=initially and default: see below},
7749 }
7750
7751 This key declares the \meta{\hologo{TeX} code} as the inner handler component
7752 of the advice.
7753
7754 The inner handler is intended to be used in situations which require
7755 knowledge of the advised command's arguments as a whole. In such situations,
7756 the outer handler will normally invoke the collector, which will in turn
7757 execute the inner handler and provide it with a single (braced) argument,
7758 containing the collected arguments of the advised command. See
7759 \refmmzauto{memoize} for the usage case which inspired this design.
7760
7761 The simplest example of an inner handler is a (single-parameter) macro which
7762 does nothing. Surprisingly enough, such an inner handler could be useful.
7763 Defining |\def\Gobble#1{}| and setting
7764 \code{\refmmz{auto}=\cs{foo}\bracestt{\refmmzauto{inner handler}=\cs{Gobble},
7765 \refmmzauto{args}=\bracestt{...}}} with the argument structure
7766 appropriate for |\foo| could be used to eradicate all invocations of |\foo|
7767 from the document.
7768
7769 The inner handler has access to all the macros available to the outer
7770 handler, but given that most of them have already fulfilled their function,
7771 only the following will likely be useful in the inner handler:
7772 \refcmd{AdviceNamespace}, \refcmd{AdviceName}, \refcmd{AdviceCsname},
7773 \refcmd{AdviceReplaced}, \refcmd{AdviceOriginal}, and \refcmd{AdviceOptions}.
7774
7775 Because there is clearly no reasonable default for the inner handler, this key
7776 is initially set to an internal control sequence producing an ``undefined
7777 inner handler'' error. Note that it is not necessary to define a dummy inner
7778 handler when handling is entirely performed by the outer handler, i.e.\ in
7779 cases when the inner handler is not invoked. Executing this key without a
7780 value restores it to the initial value.
7781
7782 During advising and advice setup, the inner handler of the advised command is
7783 accessible through \docaux{cmd}{AdviceInnerHandler}, a parameterless macro
7784 expanding to the given \meta{\hologo{TeX} code}.
7785 \end{doc}
7786
7787 \begin{doc}{easy,
7788 key={name=options,par=\marg{keylist},
7789 desc={cumulative, initially empty, value required}},
7790 key={name=clear options},
7791 }
7792
7793 The first key appends the given \meta{keylist} to the list of advice options
7794 (after prefixing it by a comma), and the second one empties this list. For a
7795 \meta{key} undefined in keypath \refkeypath{/mmz/auto},
7796 \meta{key}|=|\meta{value} has the same effect as
7797 \refmmzauto{options}|=|\bracestt{\meta{key}=\meta{value}}.
7798
7799 In Memoize, the options set by this key are known as
7800 \emph{auto-options} --- options which are applied (using \refcmd{mmzset}) at
7801 every invocation of the advised command or environment. For example, the
7802 \env{tcolorbox} environment of package \pkg{tcolorbox} is used extensively
7803 for typesetting this manual, and I have submitted this environment to
7804 automemoization. However, the \env{tcolorbox}es in this manual often include
7805 code listings. To memoize such environments successfully, their bodies must
7806 be grabbed verbatim. I have therefore submitted the \env{tcolorbox}
7807 environment to automemoization like this:
7808 \refmmz{auto}|={tcolorbox}|\bracestt{\refmmzauto{memoize},
7809 \refmmzauto{options}=\refmmz{verbatim}}; the simpler
7810 \refmmz{auto}|={tcolorbox}|\bracestt{\refmmzauto{memoize}, \refmmz{verbatim}}
7811 would work as well.
7812
7813 In general, whether to use the options set by this key, and how, remains at
7814 the sole discretion of the advice. Note that they might be used by either
7815 the outer or the inner handler, or perhaps even the collector.
7816
7817 During advising and advice setup, the options of the advised command are
7818 accessible through \docaux{cmd}{AdviceOptions}, a parameterless macro expanding
7819 to the given \meta{keylist}.
7820 \end{doc}
7821
7822 \begin{doc}{key={name=reset, desc=style}}
7823 Executing this key restores all \refmmz{auto} keys to their initial values.
7824
7825 Invoking \refmmz{auto} on the same command or environment again
7826 \emph{updates} the advice configuration. Use this key to start from scratch.
7827 \end{doc}
7828
7829 \begin{doc}{key={name=after setup, desc={initially empty, cumulative}}}
7830 The code given to this key will be executed after exiting the group opened by
7831 \refmmz{auto}. The same effect may be achieved by appending to macro
7832 \docaux{cmd}{AdviceAfterSetup}.
7833
7834 For example, \refmmzauto{integrated driver} uses this key to declare a new
7835 conditional.
7836 \end{doc}
7837
7838 \paragraph{Commands available during the execution of advice}
7839
7840 With the exception of \refcmd{AdviceGetOriginal} and
7841 \refcmd{AdviceCsnameGetOriginal}, the commands listed below only become
7842 available in the outer handler, and if that handler does nothing funky, they
7843 should be available in the collector and the inner handler, as well. However,
7844 once the advice yields control to foreign code, these macros are not guaranteed
7845 to hold the expected values anymore, because the foreign code might trigger
7846 another piece of advice. Consequently, these macros should be expanded, once,
7847 before integrating them into arbitrary (non-advice) code; in particular, this
7848 applies to \refcmd{AdviceOriginal}.
7849
7850 \begin{doc}{cmd={name=AdviceNamespace}}
7851 This macro holds the \meta{namespace}, i.e.\ the keypath which this instance
7852 of the advising framework was installed into.
7853 \end{doc}
7854
7855 \begin{doc}{cmd={name=AdviceName}}
7856 This macro holds the name of the advised command or environment, i.e.\ the
7857 name which was used as the first argument to \refmmz{auto}. For a command,
7858 this will be a control sequence, e.g.\ |\foo|; for environments (in any
7859 \hologo{TeX} format), their name, e.g.\ |foo|.
7860 \end{doc}
7861
7862 \begin{doc}{cmd={name=AdviceCsname}}
7863 This macro holds the control sequence name of the advised command; it is
7864 undefined for environments. For example, when \refcmd{AdviceName} contains
7865 |\foo@bar|, this macro will hold |foo@bar|.
7866 \end{doc}
7867
7868 \begin{doc}{cmd={name=AdviceReplaced}}
7869 This macro holds the code which was replaced by the outer handler. For
7870 commands, this will be the command itself, e.g.\ |\foo|, so
7871 \refcmd{AdviceReplaced} will equal \refcmd{AdviceName}. For an environment
7872 |foo|, \refcmd{AdviceReplaced} is set to |\begin{foo}| in \hologo{LaTeX},
7873 |\foo| in \hologo{plainTeX} and |\startfoo| in \hologo{ConTeXt}.
7874 \end{doc}
7875
7876 \begin{doc}{cmd={name=AdviceOriginal}}
7877 This macro executes the original code of the advised command.
7878
7879 This macro is defined as \refcmd{AdviceGetOriginal}\marg{namespace}\marg{name},
7880 and therefore acts as a shortcut for an explicit invocation of
7881 \refcmd{AdviceGetOriginal}. When executing the original command directly from
7882 the advice, one may safely write \refcmd{AdviceOriginal}. However, whenever
7883 \refcmd{AdviceOriginal} is embedded in code which might contain other advised
7884 commands, it should be pre-expanded, exactly once.
7885 \end{doc}
7886
7887 \begin{doc}{cmd={name=AdviceGetOriginal, par=\marg{namespace}\marg{control sequence}}}
7888 This command invokes the original definition of the \meta{control sequence}
7889 advised by the \meta{namespace} instantiation of Advice; more precisely, the
7890 full expansion of this macro produces the (internal) control sequence holding
7891 the definition of \meta{control sequence} in effect when this control
7892 sequence was activated in \meta{namespace}. This macro may be safely used
7893 outside the advice, even if the advised command is not activated.
7894
7895 For example, upon executing key |/ns/|\refmmz{auto}|=\foo{...}|,
7896 |\AdviceGetOriginal{/ns}{\foo}| will recall the original definition of |\foo|
7897 if |\foo| is activated, and simply execute |\foo| otherwise.
7898
7899 The second argument of this command should \emph{not} be an environment name.
7900 To execute the original environment |foo| in \hologo{plainTeX} or
7901 \hologo{ConTeXt}, use \refcmd{AdviceGetOriginal} with the appropriate macro:
7902 |\AdviceGetOriginal{/ns}{\foo}| or |\AdviceGetOriginal{/ns}{\startfoo}|. In
7903 \hologo{LaTeX}, one should use |\AdviceGetOriginal{/ns}{\begin}{foo}|, which
7904 executes the original \refcmd{begin} and provides it with the environment
7905 name.
7906
7907 Within the advice, you will probably never have to use this command directly,
7908 but will rather rely on the (plain or pre-expanded) \refcmd{AdviceOriginal}.
7909 However, outside the advice, this command provides the only official means to
7910 access the original definition of an advised command. (Unlike the
7911 commands described above, this command is available throughout the
7912 document.)
7913
7914 \begin{tcolorbox}[warning]
7915 A typo in the invocation of this command may result in an infinite loop.
7916 Assume that the advice for |\foo|, declared in namespace \refkeypath{/mmz},
7917 executes \refcmd{AdviceGetOriginal}|{/zzm}{\foo}|, which incorrectly refers
7918 to the non-existing namespace |/zzm|, and that command |\foo| is activated.
7919 Executing |\foo| will eventually execute
7920 \refcmd{AdviceGetOriginal}|{/zzm}{\foo}|, which won't find the original
7921 definition of |\foo| in the non-existing namespace |/zzm| and will thus
7922 execute macro |\foo| (again), which, being advised, will lead to another
7923 \refcmd{AdviceGetOriginal}|{/zzm}{\foo}|, etc. My advice is to define an
7924 abbreviation like |\def\mmzAdviceGetOriginal{\AdviceGetOriginal{/mmz}}|.
7925 And note that the name\-space is a full keypath, which begins with a slash
7926 (|/|), but has no slash at the end.
7927 \end{tcolorbox}
7928 \end{doc}
7929
7930 \begin{doc}{cmd={name=AdviceCsnameGetOriginal,
7931 par=\marg{namespace}\marg{control sequence name}}}
7932 This is a version of \refcmd{AdviceGetOriginal} which accepts a control
7933 sequence name as the second argument.
7934
7935 This macro is used by auto-key \refmmzauto{to context} to include the meaning
7936 of any command (even internal commands containing |@|, or \pkg{expl3}
7937 commands) into the context.
7938 \end{doc}
7939
7940 \paragraph{Support for specific packages}
7941 At the moment, Advice only implements specific support for \TikZ, by defining a
7942 \refmmzauto{collector} for command \refcmd{tikz}.
7943
7944 \begin{doc}{
7945 cmd={name=AdviceCollectTikZArguments},
7946 }
7947 This command collects the arguments in the format expected by \refcmd{tikz}, and
7948 executes macro \refcmd{AdviceInnerHandler} with the collected arguments given
7949 as a single braced argument. The collector supports both the group and the
7950 semicolon invocation of |\tikz|, i.e.\ both |\tikz{...}| and |\tikz...;|.
7951
7952 This command is only available upon |\input|ting file
7953 \docaux{file}{advice-tikz.code.tex}.
7954 \end{doc}
7955
7956 \endgroup % keypath = /mmz/auto
7957
7958 \subsubsection{Memoization-related additions to the advising framework}
7959 \label{sec:ref:advice:memoization}
7960
7961 In section~\ref{sec:ref:advice}, we have seen that Memoize installs the
7962 advising framework into keypath \refkeypath{/mmz}, with setup key name
7963 \refmmz{auto}. This populates keypaths \refkeypath{/mmz} and
7964 \refkeypath{/mmz/auto} with various generic advice keys. However, Memoize
7965 installs further advice\slash automemoization-related keys into these keypaths.
7966 It is these keys which are described in this section.\footnote{One such key,
7967 \refmmzauto{integrated driver}, is actually documented in
7968 section~\ref{sec:ref:memoization}.}
7969
7970 Therefore, in contrast to section~\ref{sec:ref:advice}, \refkeypath{/mmz} and
7971 \refmmz{auto} have no secret generic meaning here, i.e.\ they should \emph{not}
7972 be generalized to \meta{namespace} and \meta{setup key} of
7973 \refkey{/handlers/.install advice}.
7974
7975 \paragraph{Keys residing in \refkeypath{/mmz}}
7976
7977 \begingroup
7978 \setlength\textfloatsep{20pt plus 2pt minus 4pt}% back to the default
7979
7980 \begin{doc}{easy,
7981 key={name=manual, description={preamble-only,{ }}, conditional=false},
7982 }
7983 When this conditional is set to true, no commands are \refmmz{activate}d at
7984 the beginning of the document. The list of commands and environments advised
7985 and activated out of the box can be found in Table~\ref{tab:advised-commands}.
7986
7987 The auto-framework allows the activation to be deferred (see
7988 \refmmz{activation} and \refkey{/handlers/.install advice}) but leaves it
7989 open to the specific instance of the framework to use the deferred activation
7990 commands as it sees fit. Normally, Memoize switches to immediate activation
7991 at the end of the preamble (hook \refmmz{begindocument/before}) and issues
7992 \refmmz{activate deferred} at the beginning of the document, more precisely
7993 in hook \refmmz{begindocument/end} (afterwards, \refmmz{activate deferred} is
7994 emptied). However, when \refmmz{manual} is in effect, the deferred
7995 activation is suppressed (though it may be still carried out by the user by
7996 executing \refmmz{activate deferred}).
7997
7998 In \hologo{LaTeX}, \refmmz{manual} affects the (internal) activation of
7999 \refcmd{begin} as well, which effectively deactivates handling of all environments.
8000 \end{doc}
8001
8002 \begin{table}
8003 \centering
8004 \begin{tabularx}{\linewidth}{llX}
8005 \toprule
8006 command/environment&handler&notes\\
8007 \midrule
8008 \refcmd{begin}&custom&Only in \hologo{LaTeX}; declared by Advice.\\
8009 \refcmd{errmessage}&\refmmzauto{abort}&Not available in \hologo{LuaTeX},
8010 where better error-detection is implemented.\\
8011 \refenv{forest}&\refmmzauto{memoize}\\
8012 \refcmd{Forest}&\refmmzauto{memoize}\\
8013 \refcmd{index}&\refmmzauto{replicate}&The argument is expanded prior to replication.\\
8014 \refcmd{label}&custom&\refmmzauto{run if memoizing}; the handler globally appends
8015 \refcmd{mmzLabel}\marg{label key}\marg{current label}
8016 to register \refcmd{mmzCCMemo}.\\
8017 \refcmd{pageref}&\refmmzauto{ref}\\
8018 \refcmd[into index=false]{pdfsavepos}&\refmmzauto{abort}&Not available in \hologo{LuaTeX}.\\
8019 \refcmd{pgfsys@getposition}&\refmmzauto{abort}&Available only in \TikZ is
8020 loaded, it aborts memoization of a picture which gets accidentally marked
8021 as
8022 ``remembered''.\\
8023 \refcmd{ref}&\refmmzauto{ref}\\
8024 \refcmd[short]{savepos}&\refmmzauto{abort}&Available only in \hologo{LuaTeX}.\\
8025 \refcmd{tikz}&\refmmzauto{memoize}\\
8026 \refenv{tikzpicture}&\refmmzauto{memoize}\\
8027 \bottomrule
8028 \end{tabularx}
8029 \caption{Commands advised by Memoize}
8030 \label{tab:advised-commands}
8031 \end{table}
8032
8033 \begin{doc}{easy,
8034 key={name=ignore spaces, conditional=false},
8035 }
8036 Ignore any spaces following \emph{auto}memoized code. This key has no effect
8037 for manual memoization, i.e.\ command \refcmd{mmz} and environment
8038 \refenv{memoize}.
8039
8040 It is common practice to conclude the definition of a command by \hologo{TeX}
8041 primitive \refcmd{ignorespaces}, which consumes any following spaces, to
8042 prevent unintended blank space after the command's invocation. Automemoizing
8043 such a command disrupts this behaviour.\footnote{It is clear that
8044 \refcmd{ignorespaces} is disrupted during utilization; in this case, the
8045 original command, including the concluding \cs{ignorespaces}, is never even
8046 executed. However, the disruption also occurs during memoization, and even
8047 during regular compilation. In both cases, the memoized code is embedded
8048 in some internal Memoize code. Therefore, the original \cs{ignorespaces}
8049 does not occur directly in front of the rest of the document.} The
8050 workaround is to use this key, normally as an option in the \refmmz{auto}
8051 declaration; it will work both for automemoized macros and environments.
8052 \end{doc}
8053
8054 \paragraph{Keys residing in \refkeypath{/mmz/auto}}
8055
8056 \begingroup
8057 \yadocset{keypath=/mmz/auto}
8058
8059 \begin{doc}{easy,key={name=memoize, desc=style}}
8060 This key sets up advice which triggers memoization of the command or
8061 environment whenever it is encountered; we often refer to such a command as
8062 ``automemoized,'' or say that it was ``submitted to automemoization.''
8063
8064 An automemoized command will consume the next-options, whether memoization
8065 actually occurs or not.
8066
8067 Under the hood, this key declares both an \refmmzauto{outer handler} and an
8068 \refmmzauto{inner handler}. The outer handler opens the memoization group
8069 (so that options can be applied locally), applies the auto-options (given by
8070 \refmmzauto{options} within \refmmz{auto}) and the next-options (given by
8071 \refcmd{mmznext}) by executing \refmmzauto{apply options}, and appends the
8072 verbatim keys to \refmmzauto{collector options} if necessary. The inner
8073 handler invokes \refcmd{Memoize} (which closes the group opened by the outer
8074 handler): the first argument is \refcmd{AdviceReplaced}, expanded once and
8075 followed by the arguments of the handled command; the second argument is
8076 \refcmd{AdviceOriginal}, also expanded once and followed by the arguments of
8077 the handled command. The inner handler also makes sure that \refmmz{ignore
8078 spaces} is respected.
8079 \end{doc}
8080
8081 \begin{doc}{easy,key={name=nomemoize,desc=style}}
8082 This key installs advice which disables memoization for the space of the
8083 command or environment; we sometimes refer to such commands as
8084 ``autodisabled.''
8085
8086 This key is merely an abbreviation for \refmmzauto{noop}|,|
8087 \refmmzauto{options}|=|\refmmz{disable}. See the documentation of
8088 \refmmzauto{noop} for further details.
8089 \end{doc}
8090
8091 \begin{doc}{key={name=noop,desc=style}}
8092 This key sets up advice which does nothing.
8093
8094 Ok, not nothing at all. The installed handler applies the auto-options and
8095 the next-options by executing \refmmzauto{apply options}, and makes sure that
8096 \refmmz{verbatim} and \refmmz{ignore spaces} are respected.
8097
8098 For commands and non-\hologo{LaTeX} environments, this key declares the same
8099 outer handler as \refmmzauto{memoize}, while the inner handler merely
8100 executes the original command (respecting the potential verbatim mode),
8101 closes the group opened by the outer handler, and makes sure that
8102 \refmmz{ignore spaces} is respected.
8103
8104 For \hologo{LaTeX} environments, which open the group necessary for the local
8105 application of options themselves, this key declares an outer handler which
8106 adds the relevant code into the next hook |env/|\meta{environment
8107 name}|/begin|. There is no need to open a group, collect the environment
8108 body, or make special provisions for the verbatim mode.
8109 \end{doc}
8110
8111
8112 \begin{doc}{key={name=apply options,desc=style}}
8113 This style, used by \refmmzauto{memoize}, \refmmzauto{nomemoize} and
8114 \refmmzauto{noop} described above, installs two handlers:
8115 \begin{itemize}
8116 \item an outer handler which opens a group, applies auto-options and
8117 next-options by executing \refcmd{mmzAutoInit}, and executes the collector;
8118 and
8119 \item a bailout handler which clears the next-options.
8120 \end{itemize}
8121 \end{doc}
8122
8123
8124 \begin{doc}{cmd={name=mmzAutoInit}}
8125 This macro applies the auto-options and the next-options.
8126
8127 Additionally, if \refmmz{verbatim}, \refmmz{verb} or \refmmz{no verbatim} was
8128 previously executed, this style appends the corresponding CollArgs key
8129 (\refcollargs{verbatim}, \refcollargs{verb} or \refcollargs{no
8130 verbatim}) to \refcmd{AdviceRawCollectorOptions}. In case several of the
8131 verbatim keys were executed, the final one takes effect.
8132 \end{doc}
8133
8134
8135 \begin{doc}{easy, key={name=abort,desc=style}}
8136 This key sets up advice which aborts any ongoing memoization.
8137
8138 Under the hood, the advice merely executes \refcmd{mmzAbort} followed by
8139 \refcmd{AdviceOriginal}. The advised command does \emph{not} consume the
8140 next-options. Out of the box, we submit two control sequences to this
8141 handler:
8142 \begin{itemize}
8143 \item \refcmd{errmessage}: this allows us to detect and abort upon at least some
8144 errors.
8145 \item \refcmd[into index=false]{pdfsavepos} (in \hologo{LuaTeX},
8146 \refcmd[short]{savepos}): one common effect is that memoization of any
8147 \TikZ picture with \refkey{/tikz/remember picture} set is aborted.
8148 \end{itemize}
8149 \end{doc}
8150
8151
8152 \begin{doc}{key={name=unmemoizable,desc=style}}
8153 This key sets up advice which aborts the ongoing memoization and marks the
8154 automemoized code as unmemoizable, so that it will be henceforth compiled
8155 regularly.
8156
8157 Under the hood, the advice merely executes \refcmd{mmzUnmemoizable} followed
8158 by \refcmd{AdviceOriginal}. The advised command does \emph{not} consume the
8159 next-options.
8160
8161 Out of the box, we submit no control sequences to this advice, but it might
8162 make sense to submit \refcmd[into index=false]{pdfsavepos}\slash
8163 \refcmd[short]{savepos}. Keys \refmmzauto{abort}|=|\refcmd[short]{savepos}
8164 and \refmmzauto{unmemoizable}|=|\refcmd[short]{savepos} will most often have
8165 the same effect, as far as the author is concerned; the former was chosen as
8166 the default because it does not produce a c-memo; see
8167 \refcmd{mmzUnmemoizable} for a situation where \refmmzauto{unmemoizable} is
8168 preferred.
8169 \end{doc}
8170
8171 \begin{doc}{easy,
8172 key={name=ref,desc=style},
8173 key={name=force ref,desc=style},
8174 }
8175
8176 These keys set up advice which adds the reference key to the context
8177 expression. They are intended to be used with cross-referencing commands
8178 such as \refcmd{ref} and \refcmd{pageref}.
8179
8180 Indeed, \refcmd{ref} and \refcmd{pageref} are submitted to this advice by
8181 Memoize, with the effect that standard cross-referencing inside memoized code
8182 ``just works.'' Note that the stabilization of the document after changing
8183 the reference takes three compilation cycles, i.e.\ one cycle more than
8184 without memoization.
8185
8186 The advice set up by \refmmzauto{ref} aborts memoization if the reference
8187 key is undefined, the rationale being that the produced memo and extern would
8188 most often be useless, and could even obscure an undefined reference. The
8189 \refmmzauto{force ref} handler produces the memo and the extern even when the
8190 reference is undefined.
8191
8192 The reference produced by the advised command should be fully expandable
8193 (because it will be expanded as a part of the context expression).
8194
8195 Typically, a \refcmd{ref} command takes a single argument, the reference key.
8196 However, some packages may define a reference command which takes optional
8197 arguments, as well; in particular, the \pkg{hyperref}'s incarnation of \refcmd{ref}
8198 takes an optional star. This advice does not care: it will accept any number
8199 of any kind of optional arguments, as long as the reference key is the first
8200 braced argument following the advised command; for example, |\ref*{key}|,
8201 |\ref[opt]{key}|, |\ref*[opt]{key}| etc.\ will all be handled correctly,
8202 while |\ref{mand}{key}| will not work. Effectively, it is as if we had set
8203 \refmmzauto{args}|=lm| --- and with the same downside, namely that an
8204 unlikely unbraced single-token reference key, like |\ref k|, will not work.
8205
8206 Under the hood, these two pieces of advice pass the reference key to macros
8207 \refcmd{mmzNoRef} and \refcmd{mmzForceNoRef}, and it is these commands ---
8208 which may also be used in user-defined advice or the document itself ---
8209 which actually add the reference key to the context expression.
8210 \end{doc}
8211
8212 \begin{doc}{easy,
8213 key={name=refrange,desc=style},
8214 key={name=force refrange,desc=style},
8215 }
8216 These keys have the same function as \refmmzauto{ref} and \refmmzauto{force
8217 ref}, but they operate on reference-range commands, such as \pkg{cleveref}'s
8218 \refcmd{crefrange}, which take two arguments (the starting and the ending reference
8219 key).
8220 \end{doc}
8221
8222 \begin{doc}{easy,
8223 key={name=multiref,desc=style},
8224 key={name=force multiref,desc=style},
8225 }
8226 These keys have the same function as \refmmzauto{ref} and \refmmzauto{force
8227 ref}, but they operate on ``multireference'' commands, such as
8228 \pkg{cleveref}'s \refcmd{cref}, which allow the author to list several
8229 comma-separated reference keys in a single argument.
8230 \end{doc}
8231
8232 \begin{doc}{
8233 key={name=to context, desc=style}
8234 }
8235 This key sets up advice which appends the original meaning of the advised
8236 command to the context. It invokes \refmmzauto{run if memoizing}, so that
8237 the command is only advised during memoization.
8238
8239 It is safe to apply this key to internal commands such as commands whose name
8240 contains |@|, or \pkg{expl3} commands. It is not necessary to provide the
8241 argument structure of the advised command (using \refmmzauto{args}).
8242
8243 \end{doc}
8244
8245 \begin{doc}{
8246 key={name=replicate, desc=style}
8247 }
8248 This key sets up advice which replicates the invocation of the command in
8249 the cc-memo during memoization.
8250
8251 When using this key, it is necessary to set \refmmzauto{args} as well. For
8252 \refcmd{index}, Memoize executes
8253 \refmmz{auto}|=\index|\bracestt{\refmmzauto{args}=m, \refmmzauto{replicate}}.
8254
8255 This key takes an auto-option, \docAux{keypath=/mmz/auto/replicate,
8256 key={name=expanded}}.\footnote{This option is unrelated to Memoize's options,
8257 settable by \refcmd{mmzset}.} If given, the collected arguments will be
8258 expanded before replicating them in the cc-memo; in \hologo{LaTeX}, this
8259 expansion is |\protect|ed.
8260
8261 In \hologo{LaTeX}, Memoize submits \refcmd{index} to this handler (with
8262 expansion). Therefore, any |\index{key}| in the memoized code gets copied
8263 into the cc-memo. Effectively, indexing from within the memoized code ``just
8264 works.''
8265
8266 Note that \refcmd{label}, despite essentially requiring replication, cannot
8267 use this advice, because it needs to replicate not only the label key but
8268 \refcmd{@currentlabel} as well.
8269 \end{doc}
8270
8271 \begin{doc}{key={name=run if memoization is possible, desc=style}}
8272 Under the run conditions installed by this key, a command is only advised if
8273 Memoize is enabled but we're not already ``within Memoize,'' i.e.\ memoizing
8274 or normally compiling some code submitted to memoization. In code:
8275 \refcmd{ifmemoize}\refcmd{ifinmemoize}|\else|\refcmd{AdviceRuntrue}|\fi\fi|.
8276
8277 Internally, this key is used by \refmmzauto{memoize} and \refmmzauto{noop}.
8278 \end{doc}
8279
8280 \begin{doc}{key={name=run if memoizing, desc=style}}
8281 Under the run conditions installed by this key, a command is only advised
8282 during memoization. In code:
8283 \refcmd{ifmemoize}\refcmd{ifmemoizing}\refcmd{AdviceRuntrue}|\fi\fi|.
8284
8285 Internally, this key is used by \refmmzauto{abort}, \refmmzauto{replicate},
8286 and \refmmzauto{ref} and friends.
8287 \end{doc}
8288
8289 \endgroup % keypath = /mmz/auto
8290 \endgroup
8291
8292
8293
8294 \subsubsection{Package CollArgs}
8295 \label{sec:ref:collargs}
8296
8297 \begingroup
8298 \yadocset{keypath=/collargs}
8299
8300 \begin{doc}{cmd={
8301 name=CollectArguments,
8302 par=\oarg{options}\marg{argument specification}\marg{next-code}%
8303 \textcolor{gray}{\meta{tokens}}}}
8304
8305 This command determines the extent to which the \meta{tokens} following the
8306 the three formal arguments of the command conform to the given \meta{argument
8307 specification}, effectively splitting \meta{tokens} into \meta{argument
8308 tokens} and the \meta{rest} of the tokens, and then executes
8309 \meta{next-code} with the \meta{argument tokens} provided as a single, braced
8310 argument:
8311 \begin{center}
8312 \meta{next-code}\marg{argument tokens}\textcolor{gray}{\meta{rest}}
8313 \end{center}
8314 If the initial part of \meta{tokens} does not conform to \meta{argument
8315 specification}, \refcmd{CollectArguments} throws an error. (In this case,
8316 \meta{next-code} is not executed, and the \meta{tokens} collected until the
8317 error are thrown away.)
8318
8319 The optional \meta{options} are processed using the \pkg{pgfkeys} utility of
8320 PGF/\TikZ (see \PGFmanual{87}), with the default path set to
8321 \docaux{key path}{collargs}. The given options apply to all the arguments in
8322 \meta{argument specification}. The recognized keys are listed in the rest of
8323 the section.
8324
8325 \refcmd{CollectArguments} expects the \meta{argument specification} to
8326 conform to the \hologo{LaTeX3} argument specification format, i.e.\ it
8327 accepts the same kind of \meta{argument specification} as
8328 \refcmd{NewDocumentCommand} and friends, complete with extensions available
8329 upon loading package \pkg{xparse}.\footnote{Note that CollArgs collects the
8330 arguments by its own means. It does \emph{not} require either
8331 \hologo{LaTeX3} or \pkg{xparse} to do its job. Specifically, it may be
8332 used in \hologo{plainTeX} and \hologo{ConTeXt} as well.} Furthermore,
8333 CollArgs introduces several extensions to the
8334 \hologo{LaTeX3}\slash\pkg{xparse} argument specification
8335 format:\footnote{Collargs internally uses a dot (|.|) to delimit the argument
8336 specification from the following argument tokens. Therefore, the dot
8337 really counts as an extra argument type, in the sense that CollArgs will
8338 stop working if the dot becomes an argument type or a modifier in some
8339 future release of \hologo{LaTeX}.}
8340 \begin{itemize}
8341 \item We introduce modifier \docAux{xparse modifier={name=\&,label=amp,index
8342 annotation/.prefix={additional\ }, index annotation/.append={\
8343 (options)} }} taking a mandatory argument specifying the options to
8344 apply to the following argument in the specification. Options given here
8345 override the \meta{options} given as the optional argument.
8346 \item The environment body type \docAux{xparse type={name=b,index
8347 annotation/.append={\ (environment body)}}} may be followed by an optional
8348 \emph{braced} argument providing the name of the environment to collect.
8349 The name given here overrides the name given by the
8350 \refcollargs{environment} option.
8351 \item The number of collected ``arguments'' is unlimited.
8352 \end{itemize}
8353
8354 For convenience, the entire \hologo{LaTeX3} argument specification format,
8355 complete with \pkg{xparse} and CollArgs extensions, is summarized in the
8356 frame \vpageref{tab:argspec}. For the official documentation of the format,
8357 consult the following sources:
8358 \begin{itemize}
8359 \item The \hologo{LaTeX3} version
8360 of \href{https://www.latex-project.org/help/documentation/usrguide.pdf}
8361 {``\LaTeX\ for authors''} documents the standard argument types.
8362 \item The \pkg{xparse} manual additionally documents the uncommon\slash
8363 historical types only available upon loading package \pkg{xparse}.
8364 \end{itemize}
8365
8366 \begin{tcolorbox}[float, before float=\hfill, label=tab:argspec,
8367 title={The \hologo{LaTeX3}\slash\pkg[white]{xparse} argument specification,
8368 with CollArgs extensions}]
8369 \begin{tabularx}{\linewidth}{>{\tt}lX}
8370 \multicolumn{2}{l}{\rm\textbf{Mandatory argument types}}\\
8371 m&standard (a single token or multiple tokens in braces)\\
8372 r\meta{token$_1$}\meta{token$_2$}
8373 &delimited by \meta{token$_1$} and \meta{token$_2$} \\
8374 R\meta{token$_1$}\meta{token$_2$}\marg{default}\footnote{\label{fn:ODRE}As
8375 \refcmd{CollectArguments} does not use the arguments but only collects
8376 them, it does not care about the default values of optional arguments.
8377 Therefore, while CollArgs recognizes argument types with defaults
8378 (\docref{xparse:O}, \docref{xparse:D}, \docref{xparse:R}, and
8379 \docref{xparse:E}), they may be safely substituted by their
8380 \texttt{-NoValue-} counterparts (\docref{xparse:o}, \docref{xparse:d},
8381 \docref{xparse:r} and \docref{xparse:e}).}
8382 &delimited by \meta{token$_1$} and \meta{token$_2$}, with a default value\\
8383 v&verbatim, in the style of \cs{verb}\footnote{\refcmd{CollectArguments}
8384 can grab an argument of any type in the \refcollargs{verbatim} mode.}\\
8385 b&the body of an environment\\
8386 [1ex]\multicolumn{2}{l}{\rm\textbf{Optional argument types}}\\
8387 o&in square brackets\\
8388 O\marg{default}\footnoteagain{fn:ODRE}&in square brackets, with a default value\\
8389 d\meta{token$_1$}\meta{token$_2$}
8390 &delimited by \meta{token$_1$} and \meta{token$_2$}\\
8391 D\meta{token$_1$}\meta{token$_2$}\marg{default}\footnoteagain{fn:ODRE}
8392 &delimited by \meta{token$_1$} and \meta{token$_2$}, with a default value\\
8393 s&an optional star\\
8394 t\meta{token}&an optional \meta{token}\\
8395 e\marg{tokens}&a set of embellishments\\
8396 E\marg{tokens}\marg{defaults}\footnoteagain{fn:ODRE}
8397 &a set of embellishments, with default values\\
8398 [1ex]\multicolumn{2}{l}{\rm\textbf{Weird argument types}}\\
8399 l\footnote{\label{fn:lug}Argument types \docref{xparse:l},
8400 \docref{xparse:u} and \docref{xparse:g} are not defined by the core
8401 \hologo{LaTeX3}; they are only available upon loading package
8402 \pkg{xparse}. As noted in the main text, CollArgs does not make any
8403 difference between standard \hologo{LaTeX3} types and the \pkg{xparse}
8404 extensions; it supports them all at all times, and even in
8405 \hologo{plainTeX} and \hologo{ConTeXt}.}
8406 &a mandatory argument until the first begin-group token\\
8407 u\marg{tokens}\footnoteagain{fn:lug}&\hologo{TeX}'s delimited argument\\
8408 g\footnoteagain{fn:lug}&an optional argument inside braces\\
8409 [1ex]\multicolumn{2}{l}{\rm\textbf{Modifiers}}\\
8410 +&allow the next argument to be long\\
8411 !&disallow spaces before arguments of type \docref{xparse:d} and \docref{xparse:t}\\
8412 >\marg{processor}&process the next argument\\
8413 [1ex]\multicolumn{2}{l}{\rm\textbf{CollArgs extensions}}\\
8414 \docref{xparse:b}\marg{name}&the body of an environment with the given \meta{name}\\
8415 \docref{xparse:amp}\marg{options}&apply CollArgs options to the next argument\\
8416 \docref{xparse:amp}\docref{xparse:amp}\marg{raw options}&apply raw CollArgs options to the next argument\\
8417 \end{tabularx}
8418 \end{tcolorbox}
8419
8420 Note that in CollArgs, \docref{xparse:O}\marg{default} has the same effect as
8421 \docref{xparse:o}, and similarly for other pairs of types with and without
8422 defaults (\docref{xparse:R} and \docref{xparse:r}, \docref{xparse:D} and
8423 \docref{xparse:d}, and \docref{xparse:E} and \docref{xparse:e}). CollArgs is
8424 dedicated to collecting the argument tokens precisely as they are given: if
8425 an optional argument is missing, its default value is \emph{not} inserted
8426 among the collected arguments --- consequently, \refcmd{CollectArguments} is
8427 utterly uninterested in the default value.
8428
8429 Collection of environments automatically adapts to the format, i.e.\ given
8430 environment body name |foo|, \refcmd{CollectArguments} knows to search for
8431 \refcmd{begin}\bracestt{foo} |...| \refcmd{end}\bracestt{foo} in
8432 \hologo{LaTeX}, \cs{foo} |...| \cs{endfoo} in \hologo{plainTeX}, and
8433 \cs{startfoo} |...| \cs{stopfoo} in \hologo{ConTeXt}. For further
8434 information on environment collection, see keys \refcollargs{ignore
8435 nesting} and \refcollargs{tags}.
8436 \end{doc}
8437
8438 \begin{doc}{cmd={
8439 name=CollectArgumentsRaw,
8440 par=\marg{option-setting code}\marg{argument specification}\marg{next-code}%
8441 \textcolor{gray}{\meta{tokens}}}}
8442
8443 This command is the programmer's interface to CollArgs, intended to be used
8444 instead of \refcmd{CollectArguments} when compilation speed is an issue. The
8445 two commands only differ in how they deal with options.
8446
8447 One difference is that for \refcmd{CollectArgumentsRaw}, the options form a
8448 mandatory rather than an optional argument. More importantly, however, they
8449 do not take the form of a keylist, but should be composed out of low-level
8450 option-setting commands. Each key documented in this section has a
8451 corresponding low-level macro; these macros are listed in footnotes alongside
8452 the keys. The name of the macro starts with |\collargs| and continues with
8453 the name of the key, without spaces, each word capitalized; if the key is
8454 boolean, this convention applies to the base of the \hologo{TeX}
8455 conditional. For example,
8456 \begin{tcblisting}{listing only, listing options app={escapechar=|}}
8457 \CollectArguments[caller=\foo, tags, verbatim]|\marg{argument specification}\marg{next-code}|
8458 \end{tcblisting}
8459 is equivalent to
8460 \begin{tcblisting}{listing only, listing options app={escapechar=|}}
8461 \CollectArgumentsRaw{%
8462 \collargsCaller{\foo}%
8463 \collargsBeginTagtrue\collargsEndTagtrue
8464 \collargsVerbatim
8465 }|\marg{argument specification}\marg{next-code}|
8466 \end{tcblisting}
8467
8468 Within the option-setting code, the programmer may also deploy macro
8469 \refcmd{collargsSet}, which processes the \meta{options} in the keylist
8470 format. One idea could be to execute this macro at the end of the low-level
8471 options; this would set the ``defaults'' using the fast programmer's
8472 interface, but still allow for user customization.
8473 \end{doc}
8474
8475 \begin{doc}{
8476 cmd={name=collargsSet, par=\meta{options}}
8477 }
8478 Locally update the CollArgs configuration.
8479
8480 The \meta{options} are a comma-separated list of \meta{key}|=|\meta{value}
8481 pairs. They are processed using the \pkg{pgfkeys} utility of PGF/\TikZ (see
8482 \PGFmanual{87}), with the default path set to \refkeypath{/collargs}.
8483
8484 This key may be used either outside an invocation of
8485 \refcmd{CollectArguments}\slash \refcmd{CollectArgumentsRaw}, to set defaults
8486 or define additional styles, or within the \meta{option-setting code} in the
8487 invocation of \refcmd{CollectArgumentsRaw}.
8488 \end{doc}
8489
8490 \begin{doc}[
8491 pi={\docaux{cmd}{collargsAlias}},
8492 ]{
8493 key={name=alias, par=\marg{symbol}\marg{definition}, desc=style},
8494 }
8495 Define \meta{symbol} as an abbreviation of \marg{definition} within the
8496 \meta{argument specification} passed to \refcmd{CollectArguments}.
8497
8498 For example, Memoize's \pkg{biblatex} support allows the user to mark the
8499 citation key argument simply by preceding it by a star (|*|) by setting
8500 \code{\refcollargs{alias}=%
8501 \braces{*}\braces{\&\&\braces{\cs{mmz@biblatex@mark@citation@key}}}}, with
8502 \cs{mmz@biblatex@mark@citation@key} an internal macro which stores the
8503 contents of register \refcmd{collargsArg} into another internal macro.
8504 \end{doc}
8505
8506 \begin{doc}[
8507 pi={\docaux{cmd}{collargsCaller}},
8508 ]{
8509 key={name=caller, par=\meta{control sequence (name)},
8510 desc={no default, initially \cs{CollectArguments}}},
8511 }
8512 Set the control sequence to refer to in error messages.
8513
8514 If \meta{tokens} do not match the \meta{argument specification},
8515 \refcmd{CollectArguments} throws an error. By default, the error message
8516 contains a reference to \refcmd{CollectArguments} itself, for example %
8517 |! Argument of \CollectArguments has an extra }.| However, this might not be
8518 very informative to the author. When \refcollargs{caller}|=\cs| is in effect,
8519 the error messages will refer to the given |\cs| instead.
8520
8521 If the value of this key is not a control sequence, it is assumed to be an
8522 environment name, but as the caller must be a macro, this name will be
8523 converted into a control sequence. Setting \refcollargs{caller}|=foo| will
8524 result in error messages referencing |\foo| in \hologo{plainTeX}, |\startfoo|
8525 in \hologo{ConTeXt} and |\begin{foo}| (a single control sequence!) in
8526 \hologo{LaTeX}.
8527 \end{doc}
8528
8529 \begin{doc}[
8530 pi=\docaux{cmd}{collargsEnvironment},
8531 ]{key={name=environment, par=\meta{environment name},
8532 desc={applicable to type \docref{xparse:b}, no default, initially empty}}
8533 }
8534
8535 Set the name of the environment collected by argument type
8536 \docref{xparse:b}.
8537 \end{doc}
8538
8539 \begin{doc}[
8540 pi={\docaux{cmd}{ifcollargsBeginTag}, \docaux{cmd}{ifcollargsEndTag};
8541 \refcollargs{tags} has no corresponding low-level command}
8542 ]{
8543 key={name=begin tag, desc={applicable to type \docref{xparse:b},{ }}, conditional=false},
8544 key={name=end tag, desc={applicable to type \docref{xparse:b},{ }}, conditional=false},
8545 key={name=tags, par=\meta{boolean}, % not a conditional
8546 desc={applicable to type \docref{xparse:b}, style, default |true|}},
8547 }
8548
8549 When \refcollargs{begin tag}\slash \refcollargs{end tag} is in effect, the
8550 begin\slash end tag will be will be prepended/appended to the collected
8551 environment body. Style \refcollargs{tags} is a shortcut for setting
8552 \refcollargs{begin tag} and \refcollargs{end tag} simultaneously.
8553
8554 In \hologo{LaTeX}, using \refcollargs{tags} will thus dress up the
8555 collected body in a pair or \refcmd{begin}\marg{environment name} and
8556 \refcmd{end}\marg{environment name}. CollArgs will automatically use the tags
8557 appropriate to the format.
8558
8559 In the verbatim modes, the added tags are verbatim as well, with the detail
8560 that in \hologo{LaTeX}, there is a slight difference between the full
8561 \refcollargs{verbatim} and the partial \refcollargs{verb} mode. In the
8562 full verbatim mode, the braces surrounding the environment name are verbatim
8563 (the characters used as braces are actually determined by key
8564 \refcollargs{braces}). In the partial verbatim, as well as the
8565 non-verbatim mode, the environment name is surrounded by a pair of actual
8566 braces of category 1 and 2, regardless of which characters are of these
8567 categories in the calling code.
8568 \end{doc}
8569
8570 \begin{doc}[
8571 pi=\docaux{cmd}{ifcollargsIgnoreNesting},
8572 ]{
8573 key={name=ignore nesting, desc={applicable to type \docref{xparse:b},{ }},
8574 conditional=false},
8575 }
8576 When this key is \emph{not} in effect, CollArgs respects the hierarchical
8577 structure created by tag pairs such as \refcmd{begin}\bracestt{foo} and
8578 \refcmd{end}\bracestt{foo}. Given the situation below on the
8579 left, argument type |b{foo}| will collect everything up until the
8580 \emph{second} |\end{foo}|. Now this is what we usually want, because
8581 \hologo{LaTeX} keeps track of environment embedding as well. However, all
8582 \emph{verbatim} environments I know of, starting with the standard
8583 \hologo{LaTeX} \refenv{verbatim}, will ignore the nesting and simply scoop up
8584 everything up to the first \cs{end}|{verbatim}|. In CollArgs, we can replicate
8585 their behaviour by setting \refcollargs{ignore nesting}, as shown below on the
8586 right. (Of course we also need to set \refcollargs{verbatim} if we want to
8587 grab the environment body in the verbatim mode.)
8588
8589 \begin{tcbraster}[raster columns=2]
8590 \begin{tcblisting}{listing only, mark region={1}{5},
8591 example title=\texttt{ignore nesting=false}}
8592 ...
8593 \begin{foo}
8594 ...
8595 \end{foo}
8596 ...
8597 \end{foo}
8598 \end{tcblisting}
8599 \tcbinputexample{collargs-ignore-nesting-true}{
8600 listing only, no attachment, mark region={1}{3},
8601 example title=\texttt{ignore nesting=true}
8602 }
8603 \end{tcbraster}
8604
8605 This key applies not only to argument type \docref{xparse:b} (in either
8606 normal or verbatim mode), but also to the verbatim argument type
8607 \docref{xparse:v} and to argument types \docref{xparse:m} and
8608 \docref{xparse:g} in the \refcollargs{verbatim} (but not normal, or
8609 \refcollargs{verb}) mode. With these keys, the relevant structure markers
8610 are braces, |{| and |}|.
8611 \end{doc}
8612
8613 \begin{doc}[
8614 pi=\docaux{cmd}{ifcollargsIgnoreOtherTags},
8615 ]{
8616 key={name=ignore other tags, desc={applicable to type \docref{xparse:b},{ }},
8617 conditional=false},
8618 }
8619 In \hologo{LaTeX}, the environment tags, \refcmd{begin}\marg{name} and
8620 \refcmd{end}\marg{name}, contain braces, which retain their usual category codes
8621 in the non-verbatim and in the partial verbatim mode. Consequently, CollArgs
8622 cannot easily search for the full tags to delimit the
8623 environment.
8624
8625 When this key is \emph{not} in effect, CollArgs takes the easy path, and
8626 determines the end of the environment only by inspection of \cs{begin}s and
8627 \cs{end}s, without reference to what \meta{name} they begin or end. Only
8628 when this key \emph{is} in effect does CollArgs inspect these \meta{name}s,
8629 effectively ignoring any \cs{begin}s and \cs{end}s not followed by the name of
8630 environment being collected. The effect of absence vs.\ presence of this key
8631 is shown below, where the shaded area marks the code collected into
8632 environment |foo|.
8633
8634 \begin{tcbraster}
8635 \tcbinputexample{collargs-ignore-other-tags}(.c1){
8636 listing only, no attachment, mark region={3}{3},
8637 example title=\texttt{ignore other tags=false}}
8638 \tcbinputexample{collargs-ignore-other-tags}(.c2){
8639 listing only, no attachment, mark region={3}{5},
8640 example title=\texttt{ignore other tags=true}}
8641 \end{tcbraster}
8642
8643 This key does not have any effect the full verbatim mode, which always
8644 behaves as if this key was set to true, because braces are of category
8645 ``other'' as well. Similarly, it is as if this key was always true in
8646 \hologo{plainTeX} and \hologo{ConTeXt}, simply because environment tags in
8647 these formats don't contain braces.
8648 \end{doc}
8649
8650 \begin{doc}[
8651 pi={\raggedright
8652 \docaux{cmd}{collargsAppendPreprocessor},
8653 \docaux{cmd}{collargsPrependPreprocessor},
8654 \docaux{cmd}{collargsAppendPostprocessor},
8655 \docaux{cmd}{collargsPrependPostprocessor}},
8656 ]{
8657 key={name=append preprocessor, par=\meta{code}, desc={style, no default}},
8658 key={name=prepend preprocessor, par=\meta{code}, desc={style, no default}},
8659 key={name=append postprocessor, par=\meta{code}, desc={style, no default}},
8660 key={name=prepend postprocessor, par=\meta{code}, desc={style, no default}},
8661 }
8662 These keys declare processors which will transform the collected argument
8663 before appending it to the argument list.
8664
8665 A processor should be a macro taking no arguments, (locally) transforming the
8666 contents of token register \docaux{cmd}{collargsArg} as it sees fit.
8667
8668 A collected argument undergoes the following transformations:
8669 \begin{itemize}
8670 \item First, the argument is processed by any \emph{pre}processors, in the
8671 order indicated by |append| and |prepend|.
8672 \item Next, the processed argument is dressed up in the delimiters according
8673 to its type. For example, an optional argument of type \docref{xparse:o}
8674 will be surrounded by square brackets.
8675 \item Finally, the delimited argument is processed by any
8676 \emph{post}processors, again in the order indicated by |append| and
8677 |prepend|.
8678 \end{itemize}
8679
8680 The following example illustrates how one could go about reimplementing Bruno
8681 Le Floch ingenious package \pkg{cprotect},\footnote{This example is merely a
8682 proof of concept. For the bells and whistles which would make it useful in
8683 real life, see the documentation of \pkg{cprotect}.} which allows the
8684 author to use verbatim text in arguments. Below, we will end up with with
8685 \cs{footnote}\braces{\meta{footnote text}} replaced by |\footnote{\input
8686 _fn.tex}|, with \meta{footnote text} waiting in file |_fn.tex|. To achieve
8687 this, we define processor |\writetofile| which dumps the argument into a
8688 file, replacing it with the |\input| statement. (Of course, to allow for
8689 verbatim content in the footnote, we also have to mark the argument as
8690 \refcollargs{verbatim}. And we use \refcollargs{no delimiters} to get rid of
8691 the braces around the footnote text.)
8692
8693 \tcbinputexample{collargs-processor}{comment=\input{\examplepath.tex}}
8694 \end{doc}
8695
8696 \begin{doc}[
8697 pi={\docaux{cmd}{collargsClearPreprocessors},
8698 \docaux{cmd}{collargsClearPostprocessors}},
8699 ]{
8700 key={name=clear preprocessors}, desc=style,
8701 key={name=clear postprocessors}, desc=style,
8702 }
8703 Clear the list of pre- or post-processors.
8704 \end{doc}
8705
8706 \begin{doc}[
8707 pi={\docaux{cmd}{collargsAppendExpandablePreprocessor},
8708 \docaux{cmd}{collargsPrependExpandablePreprocessor},
8709 \docaux{cmd}{collargsAppendExpandablePostprocessor},
8710 \docaux{cmd}{collargsPrependExpandablePostprocessor}
8711 },
8712 ]{
8713 key={name=append expandable preprocessor, par=\meta{code}, desc={style, no default}},
8714 key={name=prepend expandable preprocessor, par=\meta{code}, desc={style, no default}},
8715 key={name=append expandable postprocessor, par=\meta{code}, desc={style, no default}},
8716 key={name=prepend expandable postprocessor, par=\meta{code}, desc={style, no default}},
8717 }
8718 These keys may be used to simplify the definition of processors which can
8719 arrive at the result by means of full expansion. A processor declared by one
8720 of these keys assigns the full expansion of \meta{code} to
8721 \refcmd{collargsArg}, i.e.\ it is defined as
8722 \code{\refcmd{collargsArg}\cs{expandafter}\braces{%
8723 \cs{expanded}\braces{\meta{code}}}}.
8724
8725 A processor added with one of these keys will end up among the processors
8726 declared by \refcollargs{append preprocessor} et al.
8727
8728 \meta{code} will typically contain \code{\cs{the}\refcmd{collargsArg}} to
8729 refer to the input state of the argument. For example, to add quotes around
8730 the collected argument: \code{\refcollargs{append expandable preprocessor}=%
8731 \nohyphen``\cs{the}\refcmd{collargsArg}''}. Or, to trim spaces from an
8732 argument using |\trim@spaces@noexp| from package \pkg{trimspaces}:
8733 \code{\refcollargs{append expandable preprocessor}=%
8734 \cs{expandafter}\nohyphen\cs{trim@spaces@noexp}\nohyphen\cs{expandafter}%
8735 \nohyphen\braces{\cs{the}\refcmd{collargsArg}}}. A pre-processor is the
8736 right choice in both these cases because we want to modify the argument
8737 before it is dressed up in the original delimiters.
8738
8739 Let us consider another, more involved example. As the example for
8740 \refcollargs{append preprocessor} above, it is inspired by \pkg{cprotect},
8741 but instead of writing the argument into a file and then reading it back, we
8742 rescan it using |\scantokens|. Remarks:
8743 \begin{enumerate*}[(a)]
8744 \item rescanning only happens when processing |\fbox|; \refcollargs{append
8745 expandable preprocessor} merely inserts |\ScanMultilineTokens| in front
8746 of the argument.
8747 \item |\ScanMultilineTokens| below temporarily changes the |\newlinechar| to
8748 13 to allow for linebreaks in (the source code of) the argument;
8749 \item |\endinput| is added to prevent the extra end-of-file space creeping
8750 in;
8751 \item we have used \refmmz{verb} instead of \refmmz{verbatim} so that the
8752 argument is eventually dressed up in a pair of real, category code 1 and 2,
8753 braces.
8754 \end{enumerate*}
8755
8756 \tcbinputexample{collargs-expandable-processor}{comment=\input{\examplepath.tex}}
8757 \end{doc}
8758
8759 \begin{doc}[
8760 pi=\docaux{cmd}{ifcollargsNoDelimiters}
8761 ]{
8762 key={name=no delimiters, conditional=false},
8763 }
8764 When this key is in effect, the collected argument will not be dressed up
8765 into delimiters that it was dressed up in \meta{argument tokens}. For
8766 example, an optional argument, encountered as \code{[\meta{argument}]}, will
8767 be spit out simply as \meta{argument}.
8768
8769 Any user-specified pre- or post-processing will still be applied (and usually
8770 is, when this key is used).
8771
8772 \tcbinputexample{collargs-nodelimiters}{comment=\small\input{\examplepath.tex}}
8773 \end{doc}
8774
8775 \begin{doc}[%
8776 pi=\docaux{cmd}{collargsReturn}
8777 ]{
8778 key={name=return,
8779 par=\docAux{
8780 of=key:/collargs/return,
8781 value={name=braced},
8782 text=\Alt,
8783 value={name=plain},
8784 text=\Alt,
8785 value={name=no},
8786 },
8787 desc={initially \code{braced}, no default}
8788 }
8789 }
8790 This key determines the form in which the collected arguments are passed on
8791 to \meta{next-code}:
8792 \begin{itemize}
8793 \item When set to \refcollargs{return=braced} (the default), \meta{next-code}
8794 receives the collected arguments as a single, braced argument.
8795 \item When set to \refcollargs{return=plain}, the collected arguments are
8796 simply inserted behind \meta{next-code}. Combined with other keys, this
8797 keys makes it possible to transform the collected arguments into form
8798 expected by \meta{next-code}. For example, |&{...} b| below transforms the
8799 body of an environment into a mandatory argument (after trimming the
8800 surrounding spaces); the preceding optional argument remains unmodified.
8801 Without \code{\refcollargs{return}=\refcollargs{return=plain}},
8802 |\processbody| would receive a single argument, containing both the
8803 optional argument and the transformed environment body; with
8804 \code{\refcollargs{return}=\refcollargs{return=plain}}, which makes
8805 CollArgs refrain from bracing the collected arguments, it receives an
8806 optional plus a mandatory argument.\footnote{For further examples, see the
8807 definition of \cs{mmz@remove@quotes} and advising of \cs{blx@bbl@entry}
8808 in the Memoize source.}
8809
8810 \tcbinputexample{collargs-return-plain}{comment=\small\input{\examplepath.tex}}
8811 \item When set to \refcollargs{return=no}, the collected arguments are not
8812 inserted into the input stream; \meta{next-code} can access them through
8813 \refcmd{collargsArgs}, a token register dedicated to holding the
8814 collected arguments (regardless of the value of \refcollargs{return}).
8815
8816 In conjunction with \code{\refcollargs{clear args}=false}, this setup can
8817 be useful for reiterating command \refcmd{CollectArguments}.\footnote{The
8818 idea illustrated with this example is deployed in the definition of
8819 \refmmzauto{cites} in the Memoize source.} Command
8820 |\CollectOptionalArguments| defined below collects any number of optional
8821 arguments given either in square brackets or in parentheses. To achieve
8822 this, we instruct CollArgs to neither clear \refcmd{collargsArgs} at the
8823 start of argument collection, nor insert it into the input stream once the
8824 collection is finished; we thus have to clear this token register manually
8825 at the start, and use it directly at the end. (There's a number of ways of
8826 detecting whether further arguments follow. We use a pure Collargs
8827 solution below: we define a postprocessor which sets an internal
8828 conditional to true if any argument was found, and test for that
8829 conditional in the \meta{next-code}.)
8830
8831 \tcbinputexample{collargs-return-no}{comment=\small\input{\examplepath.tex}}
8832 \end{itemize}
8833 \end{doc}
8834
8835 \begin{doc}{
8836 cmd={name=collargsArgs, desc={token register, global}},
8837 }
8838 This token register holds the collected arguments upon the exit from
8839 \refcmd{CollectArguments}; the token register is assigned to globally.
8840
8841 This register may be manipulated by the user, both within and outside an
8842 invocation of \refcmd{CollectArguments}; all assignments should be global.
8843 \end{doc}
8844
8845 \begin{doc}[
8846 pi=\docaux{cmd}{ifcollargsClearArgs}
8847 ]{
8848 key={name=clear args, conditional=true},
8849 }
8850 When this conditional is set to |false|, token register \refcmd{collargsArgs}
8851 is not cleared upon entering \refcmd{CollectArguments}; see
8852 \code{\refcollargs{return}=\refcollargs{return=no}} for an example.
8853 \end{doc}
8854
8855 \begin{doc}[
8856 pi={\docaux{cmd}{collargsVerbatim},
8857 \docaux{cmd}{collargsVerb},
8858 \docaux{cmd}{collargsNoVerbatim}.
8859 To ensure the same effect as with the keys,
8860 place these macros at the end of the option code},
8861 ]{
8862 key={name=verbatim, desc=style},
8863 key={name=verb, desc=style},
8864 key={name=no verbatim, desc={style, the initial mode}},
8865 }
8866 Select the full verbatim, the partial verbatim, or the non-verbatim mode of
8867 argument collection.
8868
8869 In the full \refcollargs{verbatim} mode, the arguments are collected under
8870 a category code regime in which all characters are of category 12, ``other''.
8871 The same goes for the partial \refcollargs{verb} mode, except that in this
8872 case, the grouping characters --- usually the braces |{| and |}| --- retain
8873 their usual category codes 1 and 2. Key \refcollargs{no verbatim} selects
8874 the normal, non-verbatim mode.
8875
8876 The partial \refcollargs{verb} mode can be useful for verbatim collection
8877 of an optional argument. To pass |]| as an optional argument to command
8878 |\foo|, we normally enclose it in braces: ˇ\foo[{~]~}]ˇ. However, if we try to
8879 collect |[{]}]| with \refcmd{CollectArguments}|[|\refmmz{verbatim}|]{o}|, we
8880 will get |{| (and most likely an error, as well), because in the
8881 \refmmz{verbatim} mode, braces do not have their grouping function. Using
8882 the \refmmz{verb} mode solves the problem: occurring within braces, the first
8883 |]| is ``invisible'' to \refcmd{CollectArguments}|[|\refmmz{verb}|]|, so the
8884 optional argument is correctly recognized as ending at the second |]|.
8885
8886 The partial \refcollargs{verb} mode is also useful for collecting the bodies
8887 of \emph{\hologo{LaTeX}} environments. The full \refcollargs{verbatim} mode
8888 will only correctly collect these bodies when the relevant \refcmd{begin}
8889 and\slash or \refcmd{end} control sequences are followed by the grouped
8890 environment name without any intervening spaces. The partial
8891 \refcollargs{verb} mode has no such restriction.
8892
8893 In the verbatim modes, modifier \docref{xparse:+} has no effect. The arguments
8894 are always collected as if they were long.
8895
8896 To correctly collect arguments in the verbatim modes, CollArgs has to mimic the
8897 many details of \hologo{TeX}'s tokenization and argument delineation. These
8898 details depend on the category code regime, and CollArgs automatically adapts
8899 to the ``outside'' category code regime, i.e.\ the regime in effect at the time
8900 of invoking \refcmd{CollectArguments}. In particular, CollArgs remembers which
8901 characters were of category codes 0, 1, 2, 5, 10 and 11, and adapts the
8902 argument collection accordingly. For example, it will correctly pick up a
8903 control sequence as a single-token \docref{xparse:m}-type (\hologo{TeX}'s undelimited)
8904 argument even when it begins with a non-standard character of category code 0.
8905 The single caveat is that only a single pair of characters can function as the
8906 grouping characters in the full verbatim mode; to compensate for the
8907 deficiency, this character pair is customizable via key
8908 \refcollargs{braces}.
8909 \end{doc}
8910
8911 \begin{doc}[
8912 pi={\docaux{cmd}{collargsFixFromVerbatim},
8913 \docaux{cmd}{collargsFixFromVerb},
8914 \docaux{cmd}{collargsFixFromNoVerbatim}},
8915 ]{
8916 key={name=fix from verbatim, desc=style},
8917 key={name=fix from verb, desc=style},
8918 key={name=fix from no verbatim, desc=style},
8919 }
8920 Key \refcollargs{fix from no verbatim} should be used when the first
8921 argument should be collected in a verbatim mode, but the outside code has
8922 already tokenized the first character of the subsequent input stream (most
8923 probably by a |\futurelet|) in the non-verbatim category code regime. Using
8924 this key will trigger a CollArgs' ``mode transition'' (described below) which
8925 will fix the situation. (This key is used in the implementation of
8926 \refcmd{mmz}.)
8927
8928 The other two keys should be used in the unlikely reverse situation, where
8929 the outside code has tokenized the following character in a verb(atim) mode,
8930 while CollArgs is requested to collect the first argument in the non-verbatim
8931 mode.
8932 \end{doc}
8933
8934 \begin{doc}[
8935 pi=\docaux{cmd}{collargsBraces}
8936 ]{
8937 key={name=braces, par={\meta{begin-group char}\meta{end-group char}},
8938 desc={no default, for the initial value see below}},
8939 }% here
8940 This key sets the verbatim begin-group and end-group characters. The setting
8941 affects collection of argument types \docref{xparse:m}, \docref{xparse:g},
8942 \docref{xparse:v}, \docref{xparse:l}, \docref{xparse:e} and (in
8943 \hologo{LaTeX}) \docref{xparse:b} in the \emph{full} verbatim
8944 mode.\footnote{The choice of the verbatim grouping characters also affects
8945 the effect of \refcollargs{begin tag} and\slash or \refcollargs{end tag};
8946 see the documentation of these keys for details.}
8947
8948 For example, in the non-verbatim and the partial verbatim mode, an
8949 \docref{xparse:m}-type argument may be delimited by any characters of
8950 category code 1 (``begin-group'') and 2 (``end-group''). In the full
8951 verbatim mode, there are of course no characters of these categories, so
8952 CollArgs internally assigns the grouping function to some pair of characters.
8953 When entering the full verbatim mode, CollArgs automatically sets the
8954 verbatim grouping characters to characters which were of categories 1 and 2
8955 in the ``outside'' category code regime, i.e.\ the regime in effect at the
8956 time of invoking \refcmd{CollectArguments}. However, in contrast to
8957 \hologo{TeX}'s internal argument parser, only one pair of characters may
8958 serve as the begin-group and the end-group character in CollArgs' full
8959 verbatim mode. In case multiple characters were of category 1 or 2 on the
8960 outside, CollArgs therefore has to make a choice, and it chooses the
8961 candidate with the lowest character code. This choice may be overridden by
8962 the user by invoking key \refcollargs{braces}; the user may even choose
8963 characters which did not belong to categories 1 and 2 in the outside regime.
8964
8965 When \meta{begin-group char} and \meta{end-group char} are of categories 1
8966 and 2 in the outside category regime, they must be enclosed in a triple
8967 group. For example, if both |()| and |{}| have the grouping function on the
8968 outside, and the user wants to select |{}| as the verbatim grouping
8969 characters (CollArgs would go for |()|, as this pair has lower character
8970 codes), the correct way to invoke this key is |braces={{{{}}}}| or
8971 |braces=((({})))|. \footnote{This complication is due to the details of
8972 \pkg{pgfkeys}' keylist processing, and does not apply to
8973 \refcmd{collargsBraces}.}
8974 \end{doc}
8975
8976 \begin{doc}[
8977 pi=\docaux{cmd}{collargsVerbatimRanges}
8978 ]{
8979 key={name=verbatim ranges,
8980 par={\bracestt{\meta{from}\texttt{-}\meta{to}(\texttt{,} \meta{from}\texttt{-}\meta{to})*}},
8981 desc={no default, initially \texttt{0-255}}},
8982 }
8983 If run under the \hologo{pdfTeX} or \hologo{XeTeX} engine, this key
8984 determines which characters will be assigned category code 12 in the verbatim
8985 mode. In \hologo{pdfTeX}, the range should remain at the initial |0-255|,
8986 but in \hologo{XeTeX}, some rare situations might require extending this
8987 range (don't attempt to set the full range of |0-1114111|, as this would be
8988 very slow and you would most likely run out of save stack).
8989
8990 In \hologo{LuaTeX}, we switch the category code regime using category code
8991 tables, so this key has another meaning: it determines the range in which
8992 CollArgs will scan for characters of category codes 1, 2 and 14, whose
8993 identity it needs to know, for internal reasons.
8994 \end{doc}
8995
8996
8997
8998 \paragraph{Mode transition limitations}
8999
9000 \refcmd{CollectArguments} has some minor limitations regarding the transition
9001 from a verbatim into non-verbatim mode, or vice versa. The gist of the issue
9002 is best illustrated with the optional argument type \docref{xparse:o} collected
9003 in the verbatim mode. CollArgs determines whether an argument of this type is
9004 present by peeking ahead (using \hologo{TeX}'s |\futurelet| primitive) into the
9005 input stream. If the argument is present (i.e.\ if the input stream continues
9006 with an open bracket, |[|%]
9007 ), all is well. But when the optional argument is absent, the peek-ahead will
9008 tokenize the following character, which presents a problem when no more
9009 arguments are present in the input stream, like in the example below, where the
9010 verbatim \docref{xparse:o} is the (only and) final type in the argument
9011 specification. In this case, the peek-ahead ``incorrectly'' assigns category
9012 code 12 (``other'') to the first |$|. %$
9013 This character was intended to be tokenized as the math shift character of
9014 category 3, to start the math mode after \refcmd{CollectArguments} is finished,
9015 but having been assigned category code 12, it cannot perform this function,
9016 resulting in error |! Missing $ inserted| %$
9017 once \hologo{TeX} encounters the superscript character |^|.
9018
9019 \tcbinputexample{collargs-transition-ok}{sidebyside, lefthand ratio=0.6,
9020 comment=\input{\examplepath.tex}}
9021
9022 Well --- this is what \emph{would} happen if CollArgs didn't address the
9023 transition issue described above. In fact, the above example compiles just
9024 fine, because CollArgs \emph{does} address this issue, but unfortunately,
9025 certain transition problems simply cannot be resolved --- read on to learn what
9026 can go wrong.
9027
9028 For example, you can typeset the name of the document author via
9029 \hologo{LaTeX}'s internal command |\@author|, but to use this command in the
9030 document, you have to precede it by |\makeatletter|. As shown by the first
9031 line of the example below, this works rather nicely: |\makeatletter| sets the
9032 category code of |@| to 11 (``letter''), so |@| may help form the control word
9033 |\@author| --- importantly, |\makeatletter| sets the category code of |@|
9034 \emph{before} control sequence |\@author| is constructed, even if it precedes
9035 it immediately.
9036
9037 \long\def\ShowArgs#1{Collected: ``{\color{red}\tt\detokenize{#1}}''}
9038 \tcbinputexample{collargs-transition-cs}{comment=\input{\examplepath.tex}}
9039
9040 In the second line of the example, our clever invocation of |\@author| is
9041 immediately preceded by a call to \refcmd{CollectArguments}, which tries to
9042 collect a verbatim argument of type \docref{xparse:o}. It doesn't find it, which results in
9043 the wrong, verbatim tokenization of the escape character of |\makeletter|.
9044 CollArgs realizes the problem and tries to fix it. But while it is searching
9045 for the end of control sequence |\makeletter| (which it successfully
9046 constructs), it triggers the tokenization of what follows --- which, as |@| is
9047 at that point of category 12 (``other''), yields the control symbol |\@| (later
9048 followed by word ``author'', typeset in the example).
9049
9050 In short, the solution has created another, delayed instance of the problem ---
9051 an instance which cannot be addressed any further. But we're nevertheless
9052 better off, as this particular issue will bite only in the case when the
9053 ``corrupted'' control sequence immediately following the invocation
9054 \refcmd{CollectArguments} changes category codes in a way that affects the
9055 tokenization of what immediately follows it.
9056
9057 This was an example of what can go wrong in the transition from the
9058 \meta{argument tokens} to the \meta{rest} of the tokens following an invocation
9059 of \refcmd{CollectArguments}. As the ``outside world'' is non-verbatim, this
9060 transition can only be problematic if the argument which corrupted the first of
9061 the \meta{rest} tokens was verbatim, so if the transition was from the verbatim
9062 to the non-verbatim mode. Such a transition can also occur within the
9063 \meta{argument tokens}, but the good news here is that CollArgs successfully
9064 solves any problems that occur there, so you should only worry about the
9065 end-of-arguments situation.
9066
9067 The other direction of the transition, from the non-verbatim to the verbatim
9068 mode, however, can affect both the internal and the external transitions. Let
9069 us illustrate the problem with the internal transition. Say you want to
9070 collect an optional argument (\docref{xparse:o}) in the non-verbatim mode, and
9071 then a mandatory argument (\docref{xparse:m}) in (either full or partial)
9072 verbatim mode. In the first invocation of \refcmd{CollectArguments} below, the
9073 optional argument is present, and we get what we expect: the percent sign is
9074 collected, verbatim, into the mandatory argument. In the second invocation,
9075 however, the percent character retains its usual commenting function ---
9076 despite the fact that we have requested verbatim mode for the mandatory
9077 argument --- which results in the group in the second line being picked up as
9078 the mandatory argument. Again, this happens because CollArgs has to peek ahead
9079 in the input stream when determining whether the optional argument is present.
9080 Having requested non-verbatim mode for the optional argument, the peeking is
9081 performed in the non-verbatim mode, and as the optional argument is not
9082 present, it finds the comment character, which fulfills its regular function of
9083 disappearing along with the rest of the line. Once CollArgs sets to find the
9084 (verbatim) mandatory argument, the rest of the line is already gone, so it
9085 searches for, and finds, this argument in the next line.
9086
9087 \long\def\ShowArgs#1{Collected: ``{\color{red}\tt\detokenize{#1}}''}
9088 \tcbinputexample{collargs-transition-comment}{listing and text}
9089
9090 Nothing can be done here --- commenting deletes information, irrevocably ---
9091 and in a similar fashion, nothing can be done to catch a verbatim end-of-line
9092 character when preceded by an absent optional argument in the non-verbatim mode
9093 (because it was already tokenized into a space token).
9094
9095 Finally, the transition issues are not limited to transits from argument type
9096 \docref{xparse:o}. The full list of argument types which give rise to
9097 transition problems (when transiting \emph{from} arguments of these types) is
9098 as follows: \docref{xparse:o}, \docref{xparse:d}, \docref{xparse:s},
9099 \docref{xparse:t}, \docref{xparse:g}, \docref{xparse:e}.
9100
9101
9102 \endgroup % /collargs
9103
9104
9105 \subsection{Support for specific classes and packages}
9106
9107 \subsubsection{PGF}
9108 \label{sec:support:pgf}
9109
9110 \PGF support defines commands \refcmd{mmzPgfAtBeginMemoization} and
9111 \refcmd{mmzPgfAtEndMemoization}, and appends them to hooks \refmmz{at begin
9112 memoization} and \refmmz{at end memoization}, respectively. This happens
9113 automatically whenever \PGF is loaded.
9114
9115 Note that environment \env{pgfpicture} is \emph{not} submitted to
9116 automemoization. Doing this would probably not be a good idea, as it is used
9117 by many other commands and environments.
9118
9119 \tcbinputexample{_support-pgf}[.excerpt]{%
9120 listing only, no .listing, no attachment,
9121 title=\hypercolor{url}{white}\PGF support,
9122 }
9123
9124 \begin{doc}{
9125 cmd={name=mmzPgfAtBeginMemoization},
9126 cmd={name=mmzPgfAtEndMemoization},
9127 }
9128 These commands are meant to be executed inside hooks \refmmz{at begin
9129 memoization} and \refmmz{at end memoization}. Together, they make sure
9130 that utilizing an extern containing PGF/\TikZ pictures advances the PGF
9131 picture ID (\cs{pgf@picture@serial@count}) as if the pictures were compiled.
9132
9133 This avoids potential problems for various mechanisms which rely on a stable
9134 PGF picture ID. For example, in absence of advancing the ID, placing a \TikZ
9135 picture scaled by |ext.scalepicture| of \pkg{tikz-ext} \emph{after} some
9136 memoized \TikZ picture would confuse the scaled picture, as its ID would
9137 depend on whether the extern of the previous (unrelated!) picture was
9138 memoized or utilized.
9139
9140 Command \refcmd{mmzPgfAtBeginMemoization}, executed in hook \refmmz{at begin
9141 memoization}, remembers the PGF picture ID at the beginning of memoization.
9142 Command \refcmd{mmzPgfAtEndMemoization}, executed in hook \refmmz{at end
9143 memoization}, checks whether the ID was advanced during memoization, and if
9144 so, appends to the cc-memo an invocation of macro
9145 \docaux{cmd}{mmzStepPgfPictureId} advancing the ID counter accordingly.
9146
9147 Note that out of the box, \refcmd{mmzPgfAtBeginMemoization} and
9148 \refcmd{mmzPgfAtEndMemoization} reside in their respective hooks at all
9149 times, not only when memoizing \refenv{tikzpicture} and \refcmd{tikz}. This
9150 way, the PGF picture ID mechanism is applied to all pictures deploying PGF
9151 pictures under the hook, e.g.\ \pkg{forest} trees.
9152 \end{doc}
9153
9154
9155 \subsubsection[TikZ]{\TikZ*}
9156
9157 When package \TikZ is loaded, environment \refenv{tikzpicture} and command
9158 \refcmd{tikz} and are automatically submitted to automemoization; this may be
9159 prevented by \code{\refmmz{deactivate}=\refenv{tikzpicture}} and\slash or
9160 \code{\refmmz{deactivate}=\refcmd{tikz}}, or by \refmmz{manual}. Note that
9161 command \refcmd{tikz} requires a dedicated argument collector,
9162 \refcmd{AdviceCollectTikZArguments}.
9163
9164 \tcbinputexample{_support-tikz}[.excerpt]{%
9165 listing only, no .listing, no attachment,
9166 title=\hypercolor{url}{white}\TikZ support,
9167 }
9168
9169
9170 \subsubsection{Forest}
9171
9172 When package \pkg{forest} is loaded, environment \refenv{forest} and command
9173 \refcmd{Forest} are automatically submitted to automemoization; this may be
9174 prevented by \code{\refmmz{deactivate}=\refenv{forest}} and\slash or
9175 \code{\refmmz{deactivate}=\refcmd{Forest}}, or by \refmmz{manual}. Note that
9176 command \refcmd{Forest} is defined by \refcmd{NewDocumentCommand}, which makes
9177 specifying option \refmmzauto{args} unnecessary.
9178
9179 \tcbinputexample{_support-forest}[.excerpt]{%
9180 listing only, no .listing, no attachment,
9181 title=\pkg[white]{Forest} support,
9182 }
9183
9184
9185 \subsubsection{Beamer}
9186
9187 In the Beamer class, Memoize cannot be loaded in the standard way, by writing
9188 |\usepackage{memoize}| in the preamble, \emph{when performing internal
9189 extraction} (the external extraction does not pose a problem). This is the
9190 case because Beamer opens the document PDF while loading the class, while
9191 Memoize requires the PDF from the previous compilation intact in order to
9192 extract the externs. The workaround is to load Memoize by writing
9193 |\RequirePackage{memoize}| \emph{before} |\documentclass{beamer}|. (The
9194 workaround is unnecessary when loading package \pkg{beamerarticle}, which runs
9195 the beamer in the |article| mode. In fact, this package may be loaded before
9196 or after Memoize.)
9197
9198 \begin{doc}{easy, key={name=per overlay,desc=style}}
9199 Applying this style to a piece of memoized code contained in a Beamer
9200 \refenv{frame} will produce a separate cc-memo (and extern) for each overlay
9201 of the frame. See section~\ref{sec:tut:beamer} for the tutorial.
9202
9203 Note that this style is \emph{not} intended for externalization of entire
9204 frames.
9205
9206 The implementation, documented in section~\ref{sec:per-overlay}, is not
9207 trivial, but only deploys the public interface offered by Memoize.
9208 \end{doc}
9209
9210 \begin{doc}{easy, key={name=beamer mode to prefix,desc=style}}
9211 Executing this key appends the beamer mode (|beamer|, |handout|, |trans| or
9212 |article|) to the \refmmz{prefix}. Use it when the externalized picture
9213 should come out different in different modes.
9214
9215 It is fine to apply the key only to selected pieces of memoized code, either
9216 via \refcmd{mmzset} or \refcmd{mmznext}.
9217
9218 Note that Beamer mode |second| is special. Under the hood, it is implemented
9219 as the default mode |beamer|. This key therefore does not facilitate having
9220 one set of externs for the single screen mode and another set for the two
9221 screen mode.
9222
9223 A simple way of executing this key for every Beamer document, and not
9224 producing an error for non-Beamer documents, is to include it in
9225 \reffile{memoize.cfg} like this: \code{beamer mode to prefix/.try}. The
9226 \code{.try} handler (see \PGFmanual{87.4.8}) works because \refmmz{beamer
9227 mode to prefix} is only defined when Beamer is loaded.
9228 \end{doc}
9229
9230 \subsubsection{Biblatex}
9231
9232 \begin{doc}{
9233 key={name=biblatex, par=\marg{additional configuration}, desc=style},
9234 }
9235 This key loads the \pkg{biblatex} support, but only if package \pkg{biblatex}
9236 is loaded (anywhere in the preamble); after loading the support,
9237 \meta{additional configuration} is executed within \refcmd{mmzset}.
9238
9239 When \pkg{biblatex} support is loaded, all citation commands defined by this
9240 package\footnote{\hbadness 3000 In \pkg{biblatex} version 3.19: \cs{cite},
9241 \cs{Cite}, \cs{parencite}, \cs{Parencite}, \cs{footcite},
9242 \cs{footcitetext}, \cs{textcite}, \cs{Textcite}, \cs{smartcite},
9243 \cs{Smartcite}, \cs{supercite}, \cs{cites}, \cs{Cites}, \cs{parencites},
9244 \cs{Parencites}, \cs{footcites}, \cs{footcitetexts}, \cs{smartcites},
9245 \cs{Smartcites}, \cs{textcites}, \cs{Textcites}, \cs{supercites},
9246 \cs{autocite}, \cs{Autocite}, \cs{autocites}, \cs{Autocites},
9247 \cs{citeauthor}, \cs{Citeauthor}, \cs{citetitle}, \cs{citeyear},
9248 \cs{citedate}, \cs{citeurl}, \cs{nocite}, \cs{fullcite}, \cs{footfullcite},
9249 \cs{volcite}, \cs{Volcite}, \cs{volcites}, \cs{Volcites}, \cs{pvolcite},
9250 \cs{Pvolcite}, \cs{pvolcites}, \cs{Pvolcites}, \cs{fvolcite},
9251 \cs{Fvolcite}, \cs{fvolcites}, \cs{Fvolcites}, \cs{ftvolcite},
9252 \cs{ftvolcites}, \cs{Ftvolcite}, \cs{Ftvolcites}, \cs{svolcite},
9253 \cs{Svolcite}, \cs{svolcites}, \cs{Svolcites}, \cs{tvolcite},
9254 \cs{Tvolcite}, \cs{tvolcites}, \cs{Tvolcites}, \cs{avolcite},
9255 \cs{Avolcite}, \cs{avolcites}, \cs{Avolcites}, \cs{notecite},
9256 \cs{Notecite}, \cs{pnotecite}, \cs{Pnotecite}, \cs{fnotecite},
9257 \cs{citename}, \cs{citelist}, \cs{citefield}.} behave like \refcmd{ref} in
9258 the sense that changing a bibliography entry referred to by a citation
9259 command within a piece of memoized code triggers recompilation of that code.
9260 For example, it is safe to use \code{\cs{textcite}\braces{\meta{citation
9261 key}}} inside memoized code, in the sense that the extern will be
9262 recompiled when the entry for the \meta{citation key} is changed in the
9263 |.bib| file.
9264
9265 Furthermore, Memoize takes care that even once an extern which contains a
9266 reference to \meta{citation key} is utilized (i.e.\ included into the
9267 document as a picture), the \meta{citation key} is marked (by \cs{nocite}) as
9268 used, and will appear among the items listed by \cs{printbibliography}.
9269
9270 In most cases, the out-of-the-box level of \refmmz{biblatex} support will
9271 suffice. However:
9272 \begin{itemize}
9273 \item To support an additional citation command (most likely defined via
9274 \cs{DeclareCiteCommand} etc.), advise it by one of \refmmzauto{cite} and
9275 friends below.
9276 \item To have ibid (and similar) tracking sensitive to citation commands
9277 within memoized code, execute \refmmz{biblatex ccmemo cite}.
9278 \end{itemize}
9279
9280 Under the hood, \pkg{biblatex} support does two things:
9281 \begin{itemize}
9282 \item It declares advice for \pkg{biblatex}'s internal macro
9283 \cs{blx@bbl@entry}, executed while processing the |.bbl| file, which makes
9284 it possible for Memoize to detect changes in bibliography entries.
9285 \item It defines the \refmmz{auto}-keys of the \refmmzauto{cite} family,
9286 documented below, and advises all \pkg{biblatex}'s citation commands with
9287 these keys.
9288 \end{itemize}
9289 \end{doc}
9290
9291
9292 \begin{doc}{
9293 key={keypath=/mmz/auto, name=cite, desc=style},
9294 key={keypath=/mmz/auto, name=cites, desc=style},
9295 key={keypath=/mmz/auto, name=volcite, desc=style},
9296 key={keypath=/mmz/auto, name=volcites, desc=style},
9297 }
9298 When a citation command is advised by one of these \refmmz{auto}-keys, a
9299 piece of memoized code containing the citation command will be recompiled
9300 upon changing the bibliography entry of a citation key used by the command;
9301 in other words, the citation command will behave in a fashion analogous to
9302 \refmmzauto{ref}.
9303
9304 Furthermore, the memo corresponding to the memoized piece of code will
9305 contain a \cs{nocite} command for each citation key occurring within the
9306 code; in effect, citation keys continue to be marked as used even when
9307 utilizing the memo\slash extern. For further details, see \refmmz{biblatex
9308 ccmemo cite}.
9309
9310 Out of the box, Memoize advises all \pkg{biblatex}'s (v3.19) citation
9311 commands with these keys. If a new citation command is defined, it may be
9312 submitted to these keys by the author. Use
9313 \begin{itemize}
9314 \item \code{\refmmz{auto}=\meta{citation command}\braces{\refmmzauto{cite}}}
9315 for a \meta{citation command} in the \cs{cite} family,
9316 \item \code{\refmmz{auto}=\meta{citation command}\braces{\refmmzauto{cites}}}
9317 for a \meta{citation command} in the \cs{cites} family,
9318 \item \code{\refmmz{auto}=\meta{citation command}\braces{\refmmzauto{volcite}}}
9319 for a \meta{citation command} in the \cs{volcite} family, and
9320 \item \code{\refmmz{auto}=\meta{citation command}\braces{\refmmzauto{volcites}}}
9321 for a \meta{citation command} in the \cs{volcites} family.
9322 \end{itemize}
9323
9324 Which key should be used with which command depends on the argument structure
9325 of the citation command, as documented in section 3.9 of the \pkg{biblatex}
9326 manual. Accordingly, the auto-keys of the \refmmzauto{cite} family set
9327 \refmmzauto{args} as follows:
9328
9329 \begin{center}
9330 \begin{tabular}{rl}
9331 &\refmmzauto{args}\\
9332 \midrule
9333 \refmmzauto{cite}, \refmmzauto{cites}
9334 &\code{l\,*m}\\
9335 \refmmzauto{volcite}, \refmmzauto{volcites}
9336 &\code{l\,m l\,*m}\\
9337 \end{tabular}
9338 \end{center}
9339
9340 Commands like |\cite| take one mandatory argument (the citation key), which
9341 may be preceded by an optional star, prenote and postnote. Key
9342 \refmmzauto{cite} grabs all the optional arguments by |l|, while |*m|
9343 receives the braced \meta{citation key} argument; |*| is locally defined to
9344 mark the following argument as the citation key. Commands like |\volcite|
9345 take two mandatory arguments: the volume and the citation key, in this order;
9346 each may be preceded by an optional argument. Again, the optional arguments
9347 are collected by |l|s, and the mandatory arguments by |m|s, with the second
9348 argument being marked as the citation key by |*|. The two \refmmzauto{cites}
9349 variants use the given \refmmzauto{args} unconventionally: they iterate the
9350 given pattern as long as the citation arguments continue (see
9351 \refcollargs{return} for the gist of how they achieve this).
9352
9353 When advising new citation commands, the author may override the given
9354 \refmmzauto{args} if necessary. In fact, this is what Memoize does do
9355 support \cs{citename}, \cs{citelist} and \cs{citefield}. With these
9356 commands, the braced \meta{citation key} is followed by optional arguments
9357 followed by another braced mandatory argument, which implies |lm| following
9358 \refmmzauto{cite}'s \refmmzauto{args}. For example, \cs{citename} is advised
9359 by \code{\refmmz{auto}=\braces{%
9360 \refmmzauto{cite},
9361 \refmmzauto{args}=l\,*m l\,m%
9362 }%
9363 }.
9364 \end{doc}
9365
9366 \begin{doc}{
9367 key={
9368 name=biblatex ccmemo cite,
9369 par={%
9370 \docAux{
9371 of=key:/mmz/biblatex ccmemo cite,
9372 value={name=nocite},
9373 text=\Alt,
9374 value={name=replicate},
9375 }
9376 },
9377 desc={initially \code{nocite}, no default}
9378 }
9379 }
9380 Select the citation command which Memoize puts into the cc-memo in order to
9381 broadcast that the citation key was used in the compilation of the extern.
9382
9383 By default, this citation command is \cs{nocite}. Even if the memoized code
9384 uses e.g.\ \cs{textcite}\braces{\meta{citation key}}, the cc-memo contains
9385 \code{\cs{nocite}\braces{\meta{citation key}}}. This usually suffices, as we
9386 normally only have to inform \pkg{biblatex} that the \meta{citation key} was
9387 used in the utilized extern.
9388
9389 However, in the rare situations where a tracker (page, cite, ibid, opcit,
9390 loccit or idem tracker) depends on the citation used in the extern, the
9391 \cs{nocite} approach does not suffice. If such tracking is required, the
9392 author should issue \code{\refmmz{biblatex ccmemo cite}=\refmmz{biblatex
9393 ccmemo cite=replicate}} to have the cc-memo contain the original citation
9394 command; note that if the cc-memo was already created, recompilation is
9395 necessary.
9396
9397 The replication approach has the downside of yielding lots of `Missing
9398 character' warnings. This happens because, in order to use the original
9399 citation command but not produce an actual citation, Memoize precedes the
9400 invocation of the original citation command by \cs{nullfont}.
9401 \end{doc}
9402
9403 \section{Varia}
9404
9405 \subsectionclearpagefalse
9406
9407
9408 \subsection{Changelog}
9409
9410 The changelogs of Memoize, Advice and CollArgs can be found in the
9411 documentation directory of the respective package, i.e.\ in
9412 \code{\meta{texmf}/doc/generic/\meta{package}/CHANGELOG.md}. They are written
9413 in the MarkDown format. Here are direct links to these changelogs on
9414 GitHub:
9415 \href{https://github.com/sasozivanovic/memoize/blob/main/CHANGELOG.md}{Memoize},
9416 \href{https://github.com/sasozivanovic/memoize/blob/main/doc/CHANGELOG.advice.md}{Advice},
9417 \href{https://github.com/sasozivanovic/memoize/blob/main/doc/CHANGELOG.collargs.md}{CollArgs}.
9418
9419 \subsection{Known issues}
9420 \label{sec:known-issues}
9421
9422 \paragraph*{Bitmap graphics export} is coming up in the next release of the
9423 package.
9424
9425
9426 \paragraph{An error occurs, but disappears in the next compilation}
9427 If a non-fatal internal \hologo{TeX} error occurs during memoization, the memos
9428 and externs may be nevertheless produced and utilized in subsequent
9429 compilations. In such a case, the erroneous code won't be compiled again, and
9430 therefore won't yield any errors, giving the mistaken impression that the code
9431 is error-free.
9432
9433 This problem does not apply to errors which trigger \refcmd{errmessage},
9434 because that control sequence is advised by \refmmzauto{abort}.
9435 Note that internal \hologo{TeX} errors like |Undefined control sequence| are
9436 \emph{not} reported through \refcmd{errmessage}, and will therefore cause the
9437 issue.
9438
9439 This problem does not affect \hologo{LuaTeX}, because this engine allows
9440 Memoize to detect errors and abort memoization if it encounters any.
9441
9442
9443 \paragraph{A minimal issue with \hologo{XeLaTeX}}
9444 If the very first page of a document of class |minimal|, compiled by
9445 \hologo{XeLaTeX}, happens to be an extern page, we have a problem: all the
9446 regular pages of the document will be of the same size as that extern page.
9447 \hologo{pdfLaTeX} and \hologo{LuaLaTeX} do not exhibit this behaviour, nor do
9448 \hologo{LaTeX} classes other than |minimal|, even when compiled with
9449 \hologo{XeLaTeX}.
9450
9451
9452 \paragraph{CollArgs}
9453
9454 Due to an unfortunate design decision, CollArgs does not accept a dot |.| as
9455 the \meta{token} argument of types, \docref{xparse:r}, \docref{xparse:R},
9456 \docref{xparse:d}, \docref{xparse:D}, and \docref{xparse:t}. The issue will be
9457 fixed in a subsequent release.
9458
9459
9460 \subsection{Troubleshooting}
9461 \label{sec:troubleshooting}
9462
9463 \paragraph{Extern extraction does not work}
9464 \label{sec:fadings-problem}
9465 \label{sec:loading-order}
9466
9467 Upon an unsuccessful extern extraction, you should get one of the errors or
9468 warnings listed below. They should appear both in the terminal output and in
9469 the log file. All but the final error can also appear when extern extraction
9470 is performed externally.
9471
9472
9473 Errors:
9474
9475 \begin{enumerate}
9476
9477 \item \code{Error: Python module 'pdfrw' was not found'},\\
9478 \code{Error: Perl module 'PDF::API' was not found"}, or\\
9479 \code{Error: Perl module 'PDF::Builder' was not found"}
9480
9481 You have not installed the PDF processing library, at least not
9482 successfully. The installation instructions can be found in
9483 section~\ref{sec:install}. Perhaps you have multiple instances of Perl\slash
9484 Python on your system and you have installed the library into the wrong
9485 instance?
9486
9487 \item\label{item:warning:extract:cannot-read} \code{Error: File
9488 "\meta{jobname}.pdf" seems corrupted. Perhaps you have to load Memoize
9489 earlier in the preamble?}
9490
9491 While the PDF could be corrupted for various reasons, the most probable
9492 reason is the package loading order, as indicated in the error message
9493 itself. (Note that the absence of \meta{jobname}|.pdf| does not yield this
9494 error. This is so that a fatal error unrelated to Memoize doesn't make
9495 Memoize throw another error in the next compilation.) Embedded extern
9496 extraction requires an intact document PDF from the previous compilation, so
9497 Memoize must be loaded before the document PDF is opened for writing the
9498 results of the ongoing compilation. In particular, the PDF is opened by \PGF
9499 library |fadings|, included by \TikZ's libraries |fadings| and |shadows|, so
9500 Memoize must be loaded before any of these libraries. With \pkg{beamer}, the
9501 problem is particularly acute because the PDF is opened while loading the
9502 class. In this case, simply moving |\usepackage{memoize}| up the preamble,
9503 as suggested, won't help: you have to write |\RequirePackage{memoize}|
9504 \emph{before} |\documentclass{beamer}|!
9505
9506 It is also possible that the PDF processing library deployed by the
9507 extraction script chokes on your particular document. Inspecting the error
9508 message original error message produced by Perl\slash Python, which can be
9509 found at the end of the long error message in the log, should help you see
9510 whether this is the case. For example, a couple of times, I got
9511 \code{%\myfontexpansionon
9512 Invalid dictionary key at /.../perl5/site\_perl/PDF/API2/Basic/PDF/File.pm
9513 line $N$\myfontexpansionoff} when extracting with the Perl-based script.
9514 The issue might be related to PDF version --- by default, \hologo{TeX}
9515 produces PDFs of version 1.5, while the PDF library
9516 \hreftt{https://metacpan.org/pod/PDF::API2}{PDF::API2} officially only
9517 supports versions up to 1.4 --- but I'm afraid I haven't identified the exact
9518 circumstances yet (possibly, the externalizing a picture containing an
9519 embedded PDF file might be the culprit). In general, the workaround is to
9520 use another extraction method or PDF processing library, see \refmmz{extract}
9521 and \refscript{memoize-extract.pl--library}.
9522
9523 \item\label{item:error:openin_any}
9524 \code{Error: I'm not allowed to write to '{\meta{filename}}' (openout\_any =
9525 {\meta{mode}})"}
9526
9527 Your \hologo{TeX} distribution does not allow this file to be written to.
9528 The relevant setting is called \docref{openout_any} in \TeXLive and
9529 |[Core]AllowUnsafeOutputFiles| in \MiKTeX.
9530
9531 \item \code{Error: I'm not allowed to read from '{\meta{filename}}' (openin\_any
9532 = {\meta{mode}})"}
9533
9534 Your \hologo{TeX} distribution does not allow this file to be read from. The
9535 relevant setting is called \docref{openin_any} in \TeXLive and
9536 |[Core]AllowUnsafeInputFiles| in \MiKTeX.
9537
9538 \item \code{Error: "Semi-absolute" paths are disallowed: '\meta{filename}'}
9539
9540 This error can only occur on Windows. The extraction scripts are even more
9541 paranoid than \hologo{TeX} and don't allow paths such as |C:foo\bar| or
9542 |\foo\bar|, neither as path to a \dmmz, memo or extern file nor as an output
9543 or temporary directory.
9544
9545 \item \code{Error: I cannot extract page {\meta{n}} '\meta{filename}.pdf', as
9546 it contains only \meta{N} pages}
9547
9548 The document PDF is too short. This could happen if you are accidentally
9549 triggering the extern extraction twice, and the first extraction was called
9550 with \refscript{memoize-extract.pl--prune}.
9551
9552 \item\label{item:error:runtime} \code{Error: Python error: \meta{error
9553 message}} or \code{Error: Perl error: \meta{error message}}
9554
9555 A runtime error occurred during the execution of the extraction script. The
9556 error message should be followed by the traceback information which includes
9557 the line number (in the extraction script) where the error occurred.
9558
9559 One possible source of this error are insufficient filesystem permissions.
9560 For example, it will occur if the current directory is marked as read-only
9561 and the temporary output directory (\texttt{\$TEXMFOUTPUT}) is not set.
9562
9563 \item \code{Error: Extraction of externs from document "\meta{jobname}.pdf"
9564 using method "\meta{extraction method}" was unsuccessful.}
9565
9566 This is a generic error produced when the extraction script was either not
9567 executed at all, or the execution didn't finish properly.\footnote{How does
9568 Memoize know whether this happened? When invoked given option
9569 \refscript{memoize-extract.pl--format}, the extraction script is supposed
9570 to write a log (actually, status) file called
9571 % todo: \mbox is a workaround preventing "font expansion: using fonts with
9572 % different limit of expansion in one paragraph is not allowed" error,
9573 % which arises in combination with "pi=..." in the CollArgs section.
9574 \mbox{\meta{jobname}}|.mmz.log|. Upon a clean completion, the final line
9575 of this log reads \cs{endinput}. If this marker is missing, Memoize
9576 produces the error under discussion. Feel free to see what's up with this
9577 file if all else fails.}
9578
9579 A couple of reasons for the failure to execute the script:
9580 \begin{itemize}
9581 \item Is the shell escape configured properly? It should be, as both
9582 Memoize's extraction scripts are listed among restricted shell escape
9583 commands in both \TeXLive and \MiKTeX, but it never hurts
9584 to check.
9585
9586 The easiest way to see if shell escape is the culprit is to compile the
9587 document with command option \docref{-shell-escape} (on \TeXLive) or
9588 \docref{--enable-write18} (on \MiKTeX); see section~\ref{sec:extraction}
9589 for details.
9590 \item Is environment variable PATH set correctly, so that the system can find
9591 the extraction script?
9592 \item Is Perl\slash Python installed on your system, and accessible to
9593 \hologo{TeX}?
9594 \end{itemize}
9595
9596 Normally, you should get error~\ref{item:error:runtime} if a runtime error
9597 occurs during the execution of the extraction script. You can only get the
9598 generic error if the runtime error occurred before the script could set up
9599 the log file, perhaps due to insufficient filesystem permissions in
9600 combination with the paranoid \docref{openout_any} setting (which is the
9601 default).
9602
9603 If the source of the error remains a mystery, I suggest inspecting the
9604 following sources of information, to help you with your investigation:
9605 \begin{itemize}
9606 \item Can you can run the extraction script by hand? Open the terminal, go
9607 into the directory containing your document, write |memoize-extract.pl
9608 |\meta{document name} (or |memoize-extract.py |\meta{document name}), and
9609 see what happens.
9610 \item Inspect \meta{jobname}|.log| --- search for
9611 \code{runsystem(memoize-extract.pl ...)}, it will tell you whether the
9612 script was executed.
9613 \item Inspect the \hologo{TeX} terminal output --- if the script was
9614 executed, it should've announced itself by \code{Extracting externs from
9615 \meta{jobname}.pdf}; are there any further messages between this header
9616 and the error message?
9617 \item Inspect \meta{the path to the extern}|.log|, if you are using
9618 \hologo{TeX}-based extraction
9619 (\code{\refmmz{extract}=\refmmz{extract=tex}}).
9620 \end{itemize}
9621
9622 \end{enumerate}
9623
9624 Not every failure to extract externs results in an error. When a warning is
9625 produced, the compilation will succeed, it's just that as Memoize cannot
9626 extract the externs, they will be produced, and dumped into your document, at
9627 each and every compilation. For warnings other than the missing document PDF,
9628 Memoize extracts as many externs as possible.
9629
9630 \begin{enumerate}
9631
9632 \item \code{Warning: Cannot open '\meta{filename}.pdf'}
9633
9634 This is not an error because what if you deleted the document PDF on purpose?
9635
9636 \item\label{item:warning:extract:unexpected-size} \code{Warning: I refuse to
9637 extract page $n$ from "\meta{jobname}.pdf", because its size
9638 ($\meta{width}\times\meta{height}$) is not what I expected\\
9639 ($\meta{expected width}\times\meta{expected height}$)}
9640
9641 If the compilation which produced the offending extern pages yielded any
9642 errors, you should probably disregard this warning, fix the errors, and
9643 compile again. Otherwise, you have somehow winded up with mismatched
9644 \meta{jobname}|.pdf| and \meta{jobname}\dmmz (the latter file contains
9645 instructions on which pages to extract, complete with the expected
9646 dimensions). Are you sure that they were produced by the same compilation,
9647 and have remained untouched since? Are you perhaps trying to perform the
9648 extraction the second time, after the first extraction
9649 \refscript{memoize-extract.pl--prune}d the PDF?
9650
9651 If the warning stubbornly persists, but you are sure that the page the script
9652 is refusing to extract is correct, you can force the extraction by adding
9653 option \refscript{memoize-extract.pl--force} to the script invocation, which
9654 can be set by \refmmz{perl extraction options}. However, as such a situation
9655 probably indicates a bug in Memoize, please let me know about it.
9656
9657 \item \code{Warning: I refuse to extract page $n$ into extern \meta{extern
9658 filename}, because the associated (c)c-memo does not exist.}
9659
9660 Assuming that you haven't deleted the memos (|.memo| files), either manually
9661 or via \refscript{memoize-clean.pl}, could it be that they were never created
9662 in the first place? Check where they should be written to (the configuration
9663 commands are listed in section~\ref{sec:ref:dirs}). Is that directory
9664 writeable, both in the sense of the system and for \hologo{TeX} (the
9665 |openout| setting in |texmf.cnf|)?
9666
9667
9668 \end{enumerate}
9669
9670
9671 % warning('I failed to execute "kpsewhich"; assuming openin_any = openout_any =
9672 % "p" '
9673
9674 \paragraph{An extern won't be included}
9675
9676 Did you receive a warning or error message?
9677 \begin{enumerate}
9678 \item \code{Package memoize Warning: Unexpected size of extern "\meta{extern
9679 path}.pdf"; expected\\ $\meta{expected width}\times\meta{expected
9680 height}$, got $\meta{width}\times\meta{height}$}
9681
9682 This warning is related to warning~\ref{item:warning:extract:unexpected-size}
9683 above, only that it occurs once the extern is extracted. The same
9684 investigative methods apply.
9685 \item \code{!pdfTeX error: pdflatex (file \meta{extern path}.pdf): reading
9686 image file failed}, or something similar for engines other than
9687 \hologo{pdfTeX}
9688
9689 This is a fatal error. The extern file got corrupted, somehow --- inexistent
9690 and even empty extern files merely trigger recompilation.
9691 \item \code{pdfTeX warning: pdflatex (file \meta{extern path}.pdf): PDF
9692 inclusion: found PDF version <$m$>, but at most version <$n$> allowed}
9693
9694 When you produced the externs, a higher \docref{reg:pdfmajorversion} and/or
9695 \docref{reg:pdfminorversion} was in effect than now. I guess you shouldn't worry
9696 about this warning if the output looks fine.
9697 \end{enumerate}
9698
9699 If there was no warning or error --- are you certain that Memoize is
9700 \refmmz{enable}d, and that it is not in the \refmmz{recompile} mode? Remember
9701 that these settings can also apply only to a part of the document; search for
9702 any stray \refcmd{mmzset} or \refcmd{mmznext} commands.
9703
9704
9705 \paragraph*{Warnings about duplicate labels, indices, etc.} may be safely
9706 disregarded.
9707
9708 Externalization causes any (non-immediate) |\write| commands in the extern to
9709 be executed twice, once upon the shipout of the regular page, and once upon the
9710 shipout of the extern page. This results in warnings about doubly defined
9711 labels, hyperreferences, indices, etc. For example, you might get
9712 \texttt{LaTeX Warning: Label `<name>' multiply defined} or \texttt{warning (pdf
9713 backend): ignoring duplicate destination with the name '<name>'}. You can
9714 safely disregard these warnings; they will disappear once the extern is
9715 utilized.
9716
9717
9718 \paragraph{Memoization was aborted}
9719
9720 This warning means that either:
9721 \begin{itemize}
9722 \item you are trying to (auto)memoize a \refenv{tikzpicture} with
9723 \refkey{/tikz/remember picture} set, or more generally, some code which
9724 contains \refcmd{savepos} --- this can't be done, see
9725 section~\ref{sec:limits}; or
9726 \item an error occurred during memoization --- in this case, Memoize cowardly
9727 refuses to proceed with memoization, see section~\ref{sec:known-issues} for
9728 details.
9729 \end{itemize}
9730
9731 \paragraph*{Memoization yields errors} while the original, non-memoized code
9732 compiles fine.
9733
9734 \ProblemsWithMemoization[in section~\ref{sec:tut:verbatim}]
9735
9736 \subsection{License}
9737
9738 Copyright \textcopyright\ 2020- Sašo Živanović.
9739
9740 This work may be distributed and/or modified under the conditions of the
9741 \hologo{LaTeX} Project Public License, either version 1.3c of this license or
9742 (at your option) any later version. The latest version of this license is
9743 in \url{https://www.latex-project.org/lppl.txt} and version 1.3 or later is
9744 part of all distributions of \hologo{LaTeX} version 2008 or later.
9745
9746 This work comprises of the sources, generated files, accompanying scripts,
9747 auxiliary files and scripts, documentation, documentation sources, examples and
9748 example sources of packages |memoize|, |nomemoize|, |memoizable|, |auto| and
9749 |collargs|. The files belonging to this work and covered by LPPL are listed in
9750 \meta{texmf}|/doc/generic/memoize/FILES|.
9751
9752 This work has the LPPL maintenance status `maintained.' The Current Maintainer
9753 of this work is Sašo Živanović. The work is available on CTAN
9754 at \url{https://ctan.org/pkg/memoize} and on GitHub
9755 at \url{https://github.com/sasozivanovic/memoize}.
9756
9757
9758 \subsection{Acknowledgments}
9759
9760 First and foremost, my gratitude goes to Stefan Müller of Language Science
9761 Press for his encouragement and patience --- without him, this package might've
9762 remained an idea forever!
9763
9764 But there were others as well. Shunsaku Hirata of the \hologo{XeTeX} team
9765 promptly resolved an issue with forward references in |xdvipdfmx|, and thereby
9766 made Memoize work with \hologo{XeTeX}. Joseph Wright kindly didn't object to
9767 me misappropriating \pkg{etoolbox} for other formats. Petra Rübe-Pugliese
9768 (CTAN) and Karl Berry (\TeXLive) offered good advice on packaging the package;
9769 I would never get to call Advice Advice without their advice. Karl also
9770 performed a security review of the extraction scripts, providing some very
9771 useful comments along the way, and agreed to include the scripts among the
9772 shell escape commands allowed in \TeXLive. Christian Schenk worked hard to
9773 support Memoize in \MiKTeX, and John Collins implemented \pkg{latexmk} support.
9774 After the package was first published, amonakov, mbert, Qrrbrbirlbel, and
9775 Ulrike Fischer provided valuable feedback, suggestions and\slash or bug
9776 reports; cfr did all that, and also advertised the package extensively.
9777
9778 Thank you all!
9779
9780
9781 \printindex
9782
9783 \end{document}
9784
9785
9786 % Local Variables:
9787 % TeX-engine: luatex
9788 % TeX-master: t
9789 % End: