Debian Codegraph @

texmf-dist/source/latex/nicetext/fifinddo.tex

raw
1 \ProvidesFile{fifinddo.tex}[2011/11/19 documenting fifinddo.sty]
2 \title{\textsf{fifinddo}\\---\\Filtering \TeX(t) Files by
3 \TeX\thanks{This file describes
4 version \fileversion\ of \textsf{\filename} as of \filedate.}}
5 % \listfiles
6 \documentclass[fleqn]{article}
7 \usepackage{makedoc}
8 % \usepackage{substr} %% for failure examples TODO 2010/03/17
9 \input{makedoc.cfg}
10 \MDkeywords{text filtering, macro programming,
11 .txt to .tex enhancement}
12 \hypersetup{%
13 pdftitle=fifinddo.sty filters text by TeX,
14 pdfsubject=documenting fifinddo.sty
15 }
16 \ReadPackageInfos{fifinddo}
17 \begin{document}
18 \maketitle
19
20 \begin{flushright}\small\it FIDO, FIND!\\or:\\FIND FIDO!\\oder:\\FIFI,
21 SUCH!\end{flushright}
22 \begin{MDabstract}
23 'fifinddo' starts implementing parsing of plain text or \TeX\ files
24 using \TeX, generalizing the philosophy behind 'docstrip',
25 based on how \TeX\ reads macro arguments.
26 Rather than typsetting the edited input stream immediately,
27 results are written to another file,
28 in the first instance as input for \TeX.
29 Rather than presenting a ``complete study" of a computer-scientific
30 idea, it aims at practical applications.
31 The main one at present is '\mbox{makedoc}' which removes certain comment
32 marks from package files and inserts listing commands.
33 Parsing macros are not defined anew at every input chunk,
34 but once before a file is processed.
35 This also allows for \emph{expandable} sequences of replacements,
36 e.g., with `txt'\,$\to$\,\TeX\ functionality.
37 The method of testing for substrings is carefully discussed,
38 revealing an earlier mistake (then) shared with 'substr.sty'
39 and \LaTeX's internal `\in@'.
40 \end{MDabstract}
41 \tableofcontents
42
43 \pagebreak %% 2011/11/19
44 \section{Introduction: The Gnome of the Aim}
45 \subsection{Parsing by \TeX---are you mad?}
46 The package name 'fifinddo' is a &\listfiles-compatible abbreviation
47 of \lq file-find-do\rq\footnote{\lq file\rq\ possibly for ``searching
48 \TeX(t) files" (I don't remember my thoughts!),
49 while there were requests for doing replacements on
50 \LaTeX\ \emph{environments} on 'texhax'. However, the package might
51 be enhanced in this direction $\dots$ so the name may be wrong
52 $\dots$ but now I like it so much .\,.\,. Or the reason was that
53 results are written to a \emph{separate file}, not typeset
54 immediately.---Let me also mention that \emph{\lq Fifi\rq} (as the
55 package name starts) is a kind of German equivalent to the
56 ``English" \emph{\lq Fido\rq}, or may have been.}
57 (or think of \lq if found do\rq).
58 'fifinddo' implements (or aims at) general parsing
59 (extracting, replacing [converting], expanding, \dots)
60 using \TeX\ where 'texhax' posters strongly urge to use 'sed',
61 'awk', or 'Perl'.
62 'fifinddo''s opposed rationales are:
63 \begin{itemize}
64 \item It works instantly on any \TeX\ installation.
65 (\emph{Restrictions:} Some \TeX\ versions &\write certain hex
66 codes for certain characters, cf.\ \TeX book p.~45,
67 I have seen this with PC\TeX. However, some applications of
68 'fifinddo' are nothing but technical steps where you will read
69 the result files rarely anyway.
70 %% And I have not yet explored extended encodings.)
71 %% <- done meanwhile 2010/03/22
72 \item You can apply and customize it like any \TeX\ macros, knowing
73 just \TeX\ (or even only the documentation of some user-friendly
74 extension of 'fifinddo'), without the need of learning any
75 additional script language.
76 \item The syntax of usual utilities (e.g., ``wildcards")
77 is sometimes difficult with \emph{\TeX} files
78 with all their backslashes, square brackets, stars,
79 question marks \dots
80 \end{itemize}
81 %
82 At least the first item is just the philosophy of the 'docstrip' program,
83 standard for installing \TeX\ packages;
84 and while I am typing this, I find at least 14 other similar packages
85 in J\"urgen Fenn's \emph{Topic Index} of the \emph{\TeX\ Catalogue:}
86 \begin{center} %% 2011/11/13
87 % \begin{quote}\small
88 % \[\texttt{%
89 % \hbox\bgroup\url{% %% \url impossible!? 2010/03/19
90 \url{%
91 http://mirror.ctan.org/help/Catalogue/bytopic.html\string#parsingfiles}%
92 % \footnotemark
93 % \egroup\]
94 % \]
95 % \footnotetext{%
96 % \href{http://mirror.ctan.org/help/Catalogue/bytopic.html\string#parsingfiles}%
97 % {Click here!}}
98 % \end{quote}
99 \end{center}
100 (Some of them may have been \emph{reactance} to 'texhax' and other
101 postings urging not to try something like this; some seem just to be
102 celebrations of the power of \TeX---yes, celebrate!)
103
104 Actually, \TeX's mechanism of collecting macro arguments is hard-wired
105 parsing at quite a high level. \LaTeX\ hides this from
106 ``simple-minded" users by a convention \emph{not} to use that full
107 power of \TeX\ for \emph{end-user macros}.
108 \emph{Internally}, \LaTeX\ \emph{does} use it in reading
109 lists of options and file dates as well as to implement certain
110 `FOR'- and `WHILE'-like loop programming structures.
111 \LaTeX's `\in@'/`\ifin@' construction is an implementation of a
112 ``<string1> occurs in <string2>" test. More packages seem to use
113 this idea for extracting file informations, like
114 'texshade'.\footnote{\url{http://ctan.org/pkg/texshade}}
115
116
117 However, such packages don't make much ado about parsing,
118 there seems to be no general setup mechanism as are presented by
119 'fifinddo'. Indeed, tayloring parsing macros to specific applications
120 may often be more efficient than a general approach.
121
122 \subsection{Useful for \dots}
123 My main application of 'fifinddo' at present is typesetting
124 documentations of packages using 'makedoc' which removes certain
125 percent marks and inserts listing commands, so you edit a package file
126 with as little documentation markup as possible.
127 This may be extended to other kinds of documents as an alternative
128 to 'easylatex' or 'wiki' (the approach of which is dangerous and
129 incompatible with certain other things).
130
131 I have used a similar own package 'txtproc' successfully,
132 where more features were implemented for practical purposes
133 than are here so far, yet I don't like its implementation, want to
134 improve it here. This package also \emph{created batch files}, e.g.,
135 to remove temporary files.
136 This could be used for package handling:
137 typset the documentation at the desired place in the tree,
138 write the packages to another, write a batch file to remove
139 files that are not needed any more after installation
140 (cf.\ 'make'). %%% !? cf. Wiki `Make (software) 2009/04/11
141
142 I used 'txtproc' also for \emph{large-scale substitutions}
143 (it had been decided to change the orthography in a part of a book).
144 Other large-scale substitutions may be:
145 \begin{itemize}
146 \item inserting &\index commands;
147 \item inserting (soft) hyphenation commands near accents;
148 \item manual umlaut-conversion.\footnote{If you know the ``names"
149 of the encodings, Heiko Oberdiek's
150 \ctanpkgref{stringenc}
151 may be preferable.}
152 \item typographical (or even orthographical) corrections (same mistake
153 many times on each of hundreds of pages).
154 You may turn `...' into `$\dots$' and `etc.' into `etc.\'
155 etc.\footnote{But what when a new sentence is starting indeed?
156 Well, `cf.' is an easier example.---`etc.' even showed
157 a problem in 'niceverb'. 'mdoccorr.cfg' replaces `etc. '
158 only, so you can keep the extra space by a code line break.}
159 This could replace packages like
160 'easylatex',\footnote{\url{http://ctan.org/pkg/easylatex}}
161 'txt2latex',\footnote{\url{http://ctan.org/pkg/txt2latex}}
162 'txt2tex'\footnote{\url{http://ctan.org/pkg/txt2tex}}
163 in a customizable way, using, e.g., the ``correct"
164 hook from 'makedoc.sty' as exemplified in 'mdoccorr.cfg'
165 (see examples section of 'makedoc.pdf').
166 You should find 'fdtxttex.tpl', a 'fifinddo' script
167 to try or apply `\MakeDocCorrectHook' from 'mdoccorr.cfg',
168 as well as 'fdtxttex.tex' that runs a dialogue for the
169 same purpose if you can manage to run it ('WinShell'?).
170 You can then try to create your own `\MakeDocCorrectHook'.
171 Section~\ref{sec:replchain} provides setup for macros
172 of this kind.
173 \item as to 'easylatex' again, \emph{lists} could be detected and
174 transformed into \LaTeX\ list commands. This could re-implement
175 the lists functionality of 'wiki.sty' that is somewhat
176 dangerous. %% TODO 2009/04/15
177 \item introduce your own \emph{shorthands} to be expanded not as \TeX\ macros,
178 but by text substitution;
179 \end{itemize}
180 %% % <- TODO present macros/"jobs" doing this.
181 %% %% <- TOOD single quote tests could be used for
182 %% double quote character as well
183 %% to turn it into left quotes if ... 2010/03/19
184 %
185 In certain cases, insertions deteriorate readability, hyphenation
186 corrections even make text search difficult.
187 It is therefore suggested to
188 \begin{enumerate}
189 \item keep editing the file without the insertions,
190 \item run the script (commands based on 'fifinddo') for insertions
191 in the preamble of the main file
192 (```\jobname.tex'", maybe &\input the script file) and
193 \item &\input the result file within the `document' environment.
194 \end{enumerate}
195
196 In general, differences to ``manual" replacing by the substitution
197 function of your \emph{text editor} is that
198 \begin{itemize}
199 \item you first keep the original version,
200 \item you can check the resulting file before you replace the original
201 file by it,
202 \item you can store the replacement script in order to check for
203 mistakes at a later stage of your work,
204 \item you can do \emph{all} the replacements in \emph{one run}
205 (by \emph{one} script to check for mistakes),
206 \item you can store replacement scripts for future applications,
207 so you needn't type the patterns and replacement strings anew.
208 \end{itemize}
209
210
211 \subsubsection{Comparisons} %% was `Missing' 2010/03/18
212 % \subsection{To implement or not to implement \dots} %% TODO!? 2009/04/13
213 % Actually, 'fifinddo' doesn't implement very much, and this possibly
214 % won't change, even if I am lucky. ``Practical applications" tend to
215 % show (in my view) that there is a very complex space of possibilities
216 % what parsing macros could do. If I try to ``attack" 'Perl' or so,
217 % cf.\ 'CPAN'\@. If I am lucky and become famous, 'fifinddo' will get many
218 % extensions in separate packages, cf.\ 'pstricks'. I also like it when
219 % you can choose among packages for similar purposes, and this may apply
220 % to extensions of 'fifinddo' as well.
221 %
222 % What \emph{really} is missing very badly at present is matching
223 % \emph{sequences} of strings. However, 'makedoc' contains a practical
224 % application of this for the case that both strings appear \emph{in the same
225 % line}. A better algorithm would have to read more lines from the input
226 % file etc.
227 %
228 % As it just comes to my frightened mind, there is even a problem with
229 % ``strings"/``patterns" that are \emph{phrases} of words, containing
230 % spaces \dots
231 %
232 % Clearly much comfort concerning \emph{routine} jobs is missing now,
233 % 'txtproc' has more. Please sponsor (or otherwise support) the work for
234 % improving this.
235 %
236 It should be noted (perhaps here) that the present approach to parsing
237 is a quite \emph{simple} one and in this respect much different to the
238 string handling mechanisms of
239 'stringstrings',\footnote{\url{http://ctan.org/pkg/stringstrings}}
240 'ted',\footnote{\url{http://ctan.org/pkg/ted}}
241 'xstrings'\footnote{\url{http://ctan.org/pkg/xstrings}}
242 (as I understand them, perhaps also
243 'coolstr'\footnote{\url{http://ctan.org/pkg/coolstr}})
244 which are \emph{much more powerful} than what is offered here---but
245 perhaps slow and for practical applications possibly replaceable by
246 the present approach.
247 \emph{Expandable replacement} seems not to exist outside 'fifinddo'
248 (2009/04/13).
249
250 Much is missing, I know.\footnote{There
251 is more in my badly implemented 'txtproc.sty'.}
252 I am just implementing what I actually need
253 and what could show that this approach is worth being pursued.
254 %% removed 2010/03/18:
255 % It may need being sponsored or otherwise supported.
256
257 \subsection{For insiders}
258
259 \emph{Warning:} You may (at least at the present state of the work)
260 have little success with this package, if you don't know about
261 \TeX's category codes and how \TeX\ macros are defined.
262 The package rather provides tools for package writers.
263 You may, however, be able to run other packages which just load
264 'fifinddo' as required background.
265
266 That 'fifinddo' acts on ``\TeX(t)" files or so means that
267 (at present) I think of applications on ``plain text" files which
268 will usually be \TeX\ input files. ``At present" they are read
269 without ``special characters," so essentially category codes of input
270 characters are either 11 (``letter") or 12 (``other").
271 This way some things are easier than with usual \TeX\ applications:
272 \begin{enumerate}
273 \item You can ``look into" curly braces and ``behind" comment characters.
274 \item There are exact or safe tests especially of \emph{empty macro arguments}
275 that are ``expandable," i.e., they are ``robust,"
276 don't need assignments, can be executed in `\write'ing and in
277 `\edef' definitions. ``Usually," the safe way to test emptiness
278 is storing a macro argument as a macro, say `\tempo', in order
279 to test `\ifx\tempo\empty' where `\empty' has been defined
280 by `\def\empty{}' in the format. But this requires some
281 `\def\tempo{#<n>}' which breaks in ``mere expanding"
282 (\TeX\ \emph{evaluates} `\tempo' instead of defining it).
283 An \emph{expandable} test on emptiness is, e.g. `\ifx$#<n>$',
284 where we hope that it becomes `\iftrue' just if macro argument
285 `#<n>' is empty indeed. However, ``usually" it may \emph{also}
286 become `\iftrue' when `#<n>' starts with `$'---if the latter
287 has category code 3 (``math shift"). But 'fifinddo' does
288 not assign category code 3 to any character from the input file!
289 Therefore `\ifx$#<n>$' is `\iftrue' \emph{exactly} if `#<n>' is
290 empty.
291 \item You can avoid interference with packages that are needed for
292 typesetting. You can do the ``preprocessing" in one run with
293 typesetting, but you should do the preprocessing before you
294 load packages needed for typesetting. One may even try to keep
295 the macros and settings for preprocessing local to a group.
296 \end{enumerate}
297 %% done, moving to .sty 2010/04/06:
298 % Once there may be an option to read input with some usual \TeX\
299 % category codes as well, it may be useful to (some of)
300 % \begin{itemize}
301 % \item avoid matching substrings of control words,
302 % \item skip blank spaces as \TeX\ does it usually,
303 % \item catch \emph{balanced} input pieces,
304 % \item ignore comments,
305 % \item ignore certain characters.
306 % \end{itemize}
307
308 The essential approach of 'fifinddo' to looking for single strings is
309 described in some detail in section~\ref{sec:theory}.
310
311 The implementation of 'fifinddo' is as follows.
312 User commands are specially highlighted (boxed\slash coloured),
313 together with their syntax description.
314
315 % \end{document}
316
317 \section{Preliminaries}
318 \subsection{Head of file (Legalese)}
319 \sloppy
320 \ProcessLineMessage{}
321 \MakeInputJobDoc{23}{\SectionLevelTwoParseInput}
322 \end{document}
323
324 TODO cf. Kastrup 'makematch' and others 2009/04/09,
325 'parselines' 2010/11/13
326
327 2009/04/10 title break, by -> with(!?); `boxed/coloured'
328 2009/04/12 useful: own shorthands
329 2009/04/13 substr.sty, \emph{expandable}, w/o `implement'!?
330 2009/04/15 same with \in@
331 2010/03/08 EPS application removed
332 2010/03/17 use \MakeInputJobDoc
333 2010/03/18 `Missing' -> `Comparisons' etc.; \Require...
334 2010/03/19 Catalogue \url; '' -> "
335 2010/03/20 \ctanpkgref
336 2010/03/22 debugging; done ...
337 2010/03/23 more URLs; abstract "then shared";
338 removed dates from 2009, fdtxttex.tex/tpl
339 2010/04/06 possible uses of reading TeX category codes moved to .sty
340 2010/11/11 \thanks <- package version
341 2010/11/12 package date updated, \author -> makedoc.cfg
342 2010/11/13 package date updated; parselines
343 2010/11/24 debugging
344 2010/11/25f. using readprov.sty
345 2010/11/27 reworked \ProvidesFile
346 2011/08/22 PDF metadata; adjusting to makedoc.cfg
347 2011/11/13 [fleqn]
348 2011/11/19 \pagebreak