| 1 |
\ProvidesFile{fifinddo.tex}[2011/11/19 documenting fifinddo.sty] |
| 2 |
\title{\textsf{fifinddo}\\---\\Filtering \TeX(t) Files by |
| 3 |
\TeX\thanks{This file describes |
| 4 |
version \fileversion\ of \textsf{\filename} as of \filedate.}} |
| 5 |
% \listfiles |
| 6 |
\documentclass[fleqn]{article} |
| 7 |
\usepackage{makedoc} |
| 8 |
% \usepackage{substr} %% for failure examples TODO 2010/03/17 |
| 9 |
\input{makedoc.cfg} |
| 10 |
\MDkeywords{text filtering, macro programming, |
| 11 |
.txt to .tex enhancement} |
| 12 |
\hypersetup{% |
| 13 |
pdftitle=fifinddo.sty filters text by TeX, |
| 14 |
pdfsubject=documenting fifinddo.sty |
| 15 |
} |
| 16 |
\ReadPackageInfos{fifinddo} |
| 17 |
\begin{document} |
| 18 |
\maketitle |
| 19 |
|
| 20 |
\begin{flushright}\small\it FIDO, FIND!\\or:\\FIND FIDO!\\oder:\\FIFI, |
| 21 |
SUCH!\end{flushright} |
| 22 |
\begin{MDabstract} |
| 23 |
'fifinddo' starts implementing parsing of plain text or \TeX\ files |
| 24 |
using \TeX, generalizing the philosophy behind 'docstrip', |
| 25 |
based on how \TeX\ reads macro arguments. |
| 26 |
Rather than typsetting the edited input stream immediately, |
| 27 |
results are written to another file, |
| 28 |
in the first instance as input for \TeX. |
| 29 |
Rather than presenting a ``complete study" of a computer-scientific |
| 30 |
idea, it aims at practical applications. |
| 31 |
The main one at present is '\mbox{makedoc}' which removes certain comment |
| 32 |
marks from package files and inserts listing commands. |
| 33 |
Parsing macros are not defined anew at every input chunk, |
| 34 |
but once before a file is processed. |
| 35 |
This also allows for \emph{expandable} sequences of replacements, |
| 36 |
e.g., with `txt'\,$\to$\,\TeX\ functionality. |
| 37 |
The method of testing for substrings is carefully discussed, |
| 38 |
revealing an earlier mistake (then) shared with 'substr.sty' |
| 39 |
and \LaTeX's internal `\in@'. |
| 40 |
\end{MDabstract} |
| 41 |
\tableofcontents |
| 42 |
|
| 43 |
\pagebreak %% 2011/11/19 |
| 44 |
\section{Introduction: The Gnome of the Aim} |
| 45 |
\subsection{Parsing by \TeX---are you mad?} |
| 46 |
The package name 'fifinddo' is a &\listfiles-compatible abbreviation |
| 47 |
of \lq file-find-do\rq\footnote{\lq file\rq\ possibly for ``searching |
| 48 |
\TeX(t) files" (I don't remember my thoughts!), |
| 49 |
while there were requests for doing replacements on |
| 50 |
\LaTeX\ \emph{environments} on 'texhax'. However, the package might |
| 51 |
be enhanced in this direction $\dots$ so the name may be wrong |
| 52 |
$\dots$ but now I like it so much .\,.\,. Or the reason was that |
| 53 |
results are written to a \emph{separate file}, not typeset |
| 54 |
immediately.---Let me also mention that \emph{\lq Fifi\rq} (as the |
| 55 |
package name starts) is a kind of German equivalent to the |
| 56 |
``English" \emph{\lq Fido\rq}, or may have been.} |
| 57 |
(or think of \lq if found do\rq). |
| 58 |
'fifinddo' implements (or aims at) general parsing |
| 59 |
(extracting, replacing [converting], expanding, \dots) |
| 60 |
using \TeX\ where 'texhax' posters strongly urge to use 'sed', |
| 61 |
'awk', or 'Perl'. |
| 62 |
'fifinddo''s opposed rationales are: |
| 63 |
\begin{itemize} |
| 64 |
\item It works instantly on any \TeX\ installation. |
| 65 |
(\emph{Restrictions:} Some \TeX\ versions &\write certain hex |
| 66 |
codes for certain characters, cf.\ \TeX book p.~45, |
| 67 |
I have seen this with PC\TeX. However, some applications of |
| 68 |
'fifinddo' are nothing but technical steps where you will read |
| 69 |
the result files rarely anyway. |
| 70 |
%% And I have not yet explored extended encodings.) |
| 71 |
%% <- done meanwhile 2010/03/22 |
| 72 |
\item You can apply and customize it like any \TeX\ macros, knowing |
| 73 |
just \TeX\ (or even only the documentation of some user-friendly |
| 74 |
extension of 'fifinddo'), without the need of learning any |
| 75 |
additional script language. |
| 76 |
\item The syntax of usual utilities (e.g., ``wildcards") |
| 77 |
is sometimes difficult with \emph{\TeX} files |
| 78 |
with all their backslashes, square brackets, stars, |
| 79 |
question marks \dots |
| 80 |
\end{itemize} |
| 81 |
% |
| 82 |
At least the first item is just the philosophy of the 'docstrip' program, |
| 83 |
standard for installing \TeX\ packages; |
| 84 |
and while I am typing this, I find at least 14 other similar packages |
| 85 |
in J\"urgen Fenn's \emph{Topic Index} of the \emph{\TeX\ Catalogue:} |
| 86 |
\begin{center} %% 2011/11/13 |
| 87 |
% \begin{quote}\small |
| 88 |
% \[\texttt{% |
| 89 |
% \hbox\bgroup\url{% %% \url impossible!? 2010/03/19 |
| 90 |
\url{% |
| 91 |
http://mirror.ctan.org/help/Catalogue/bytopic.html\string#parsingfiles}% |
| 92 |
% \footnotemark |
| 93 |
% \egroup\] |
| 94 |
% \] |
| 95 |
% \footnotetext{% |
| 96 |
% \href{http://mirror.ctan.org/help/Catalogue/bytopic.html\string#parsingfiles}% |
| 97 |
% {Click here!}} |
| 98 |
% \end{quote} |
| 99 |
\end{center} |
| 100 |
(Some of them may have been \emph{reactance} to 'texhax' and other |
| 101 |
postings urging not to try something like this; some seem just to be |
| 102 |
celebrations of the power of \TeX---yes, celebrate!) |
| 103 |
|
| 104 |
Actually, \TeX's mechanism of collecting macro arguments is hard-wired |
| 105 |
parsing at quite a high level. \LaTeX\ hides this from |
| 106 |
``simple-minded" users by a convention \emph{not} to use that full |
| 107 |
power of \TeX\ for \emph{end-user macros}. |
| 108 |
\emph{Internally}, \LaTeX\ \emph{does} use it in reading |
| 109 |
lists of options and file dates as well as to implement certain |
| 110 |
`FOR'- and `WHILE'-like loop programming structures. |
| 111 |
\LaTeX's `\in@'/`\ifin@' construction is an implementation of a |
| 112 |
``<string1> occurs in <string2>" test. More packages seem to use |
| 113 |
this idea for extracting file informations, like |
| 114 |
'texshade'.\footnote{\url{http://ctan.org/pkg/texshade}} |
| 115 |
|
| 116 |
|
| 117 |
However, such packages don't make much ado about parsing, |
| 118 |
there seems to be no general setup mechanism as are presented by |
| 119 |
'fifinddo'. Indeed, tayloring parsing macros to specific applications |
| 120 |
may often be more efficient than a general approach. |
| 121 |
|
| 122 |
\subsection{Useful for \dots} |
| 123 |
My main application of 'fifinddo' at present is typesetting |
| 124 |
documentations of packages using 'makedoc' which removes certain |
| 125 |
percent marks and inserts listing commands, so you edit a package file |
| 126 |
with as little documentation markup as possible. |
| 127 |
This may be extended to other kinds of documents as an alternative |
| 128 |
to 'easylatex' or 'wiki' (the approach of which is dangerous and |
| 129 |
incompatible with certain other things). |
| 130 |
|
| 131 |
I have used a similar own package 'txtproc' successfully, |
| 132 |
where more features were implemented for practical purposes |
| 133 |
than are here so far, yet I don't like its implementation, want to |
| 134 |
improve it here. This package also \emph{created batch files}, e.g., |
| 135 |
to remove temporary files. |
| 136 |
This could be used for package handling: |
| 137 |
typset the documentation at the desired place in the tree, |
| 138 |
write the packages to another, write a batch file to remove |
| 139 |
files that are not needed any more after installation |
| 140 |
(cf.\ 'make'). %%% !? cf. Wiki `Make (software) 2009/04/11 |
| 141 |
|
| 142 |
I used 'txtproc' also for \emph{large-scale substitutions} |
| 143 |
(it had been decided to change the orthography in a part of a book). |
| 144 |
Other large-scale substitutions may be: |
| 145 |
\begin{itemize} |
| 146 |
\item inserting &\index commands; |
| 147 |
\item inserting (soft) hyphenation commands near accents; |
| 148 |
\item manual umlaut-conversion.\footnote{If you know the ``names" |
| 149 |
of the encodings, Heiko Oberdiek's |
| 150 |
\ctanpkgref{stringenc} |
| 151 |
may be preferable.} |
| 152 |
\item typographical (or even orthographical) corrections (same mistake |
| 153 |
many times on each of hundreds of pages). |
| 154 |
You may turn `...' into `$\dots$' and `etc.' into `etc.\' |
| 155 |
etc.\footnote{But what when a new sentence is starting indeed? |
| 156 |
Well, `cf.' is an easier example.---`etc.' even showed |
| 157 |
a problem in 'niceverb'. 'mdoccorr.cfg' replaces `etc. ' |
| 158 |
only, so you can keep the extra space by a code line break.} |
| 159 |
This could replace packages like |
| 160 |
'easylatex',\footnote{\url{http://ctan.org/pkg/easylatex}} |
| 161 |
'txt2latex',\footnote{\url{http://ctan.org/pkg/txt2latex}} |
| 162 |
'txt2tex'\footnote{\url{http://ctan.org/pkg/txt2tex}} |
| 163 |
in a customizable way, using, e.g., the ``correct" |
| 164 |
hook from 'makedoc.sty' as exemplified in 'mdoccorr.cfg' |
| 165 |
(see examples section of 'makedoc.pdf'). |
| 166 |
You should find 'fdtxttex.tpl', a 'fifinddo' script |
| 167 |
to try or apply `\MakeDocCorrectHook' from 'mdoccorr.cfg', |
| 168 |
as well as 'fdtxttex.tex' that runs a dialogue for the |
| 169 |
same purpose if you can manage to run it ('WinShell'?). |
| 170 |
You can then try to create your own `\MakeDocCorrectHook'. |
| 171 |
Section~\ref{sec:replchain} provides setup for macros |
| 172 |
of this kind. |
| 173 |
\item as to 'easylatex' again, \emph{lists} could be detected and |
| 174 |
transformed into \LaTeX\ list commands. This could re-implement |
| 175 |
the lists functionality of 'wiki.sty' that is somewhat |
| 176 |
dangerous. %% TODO 2009/04/15 |
| 177 |
\item introduce your own \emph{shorthands} to be expanded not as \TeX\ macros, |
| 178 |
but by text substitution; |
| 179 |
\end{itemize} |
| 180 |
%% % <- TODO present macros/"jobs" doing this. |
| 181 |
%% %% <- TOOD single quote tests could be used for |
| 182 |
%% double quote character as well |
| 183 |
%% to turn it into left quotes if ... 2010/03/19 |
| 184 |
% |
| 185 |
In certain cases, insertions deteriorate readability, hyphenation |
| 186 |
corrections even make text search difficult. |
| 187 |
It is therefore suggested to |
| 188 |
\begin{enumerate} |
| 189 |
\item keep editing the file without the insertions, |
| 190 |
\item run the script (commands based on 'fifinddo') for insertions |
| 191 |
in the preamble of the main file |
| 192 |
(```\jobname.tex'", maybe &\input the script file) and |
| 193 |
\item &\input the result file within the `document' environment. |
| 194 |
\end{enumerate} |
| 195 |
|
| 196 |
In general, differences to ``manual" replacing by the substitution |
| 197 |
function of your \emph{text editor} is that |
| 198 |
\begin{itemize} |
| 199 |
\item you first keep the original version, |
| 200 |
\item you can check the resulting file before you replace the original |
| 201 |
file by it, |
| 202 |
\item you can store the replacement script in order to check for |
| 203 |
mistakes at a later stage of your work, |
| 204 |
\item you can do \emph{all} the replacements in \emph{one run} |
| 205 |
(by \emph{one} script to check for mistakes), |
| 206 |
\item you can store replacement scripts for future applications, |
| 207 |
so you needn't type the patterns and replacement strings anew. |
| 208 |
\end{itemize} |
| 209 |
|
| 210 |
|
| 211 |
\subsubsection{Comparisons} %% was `Missing' 2010/03/18 |
| 212 |
% \subsection{To implement or not to implement \dots} %% TODO!? 2009/04/13 |
| 213 |
% Actually, 'fifinddo' doesn't implement very much, and this possibly |
| 214 |
% won't change, even if I am lucky. ``Practical applications" tend to |
| 215 |
% show (in my view) that there is a very complex space of possibilities |
| 216 |
% what parsing macros could do. If I try to ``attack" 'Perl' or so, |
| 217 |
% cf.\ 'CPAN'\@. If I am lucky and become famous, 'fifinddo' will get many |
| 218 |
% extensions in separate packages, cf.\ 'pstricks'. I also like it when |
| 219 |
% you can choose among packages for similar purposes, and this may apply |
| 220 |
% to extensions of 'fifinddo' as well. |
| 221 |
% |
| 222 |
% What \emph{really} is missing very badly at present is matching |
| 223 |
% \emph{sequences} of strings. However, 'makedoc' contains a practical |
| 224 |
% application of this for the case that both strings appear \emph{in the same |
| 225 |
% line}. A better algorithm would have to read more lines from the input |
| 226 |
% file etc. |
| 227 |
% |
| 228 |
% As it just comes to my frightened mind, there is even a problem with |
| 229 |
% ``strings"/``patterns" that are \emph{phrases} of words, containing |
| 230 |
% spaces \dots |
| 231 |
% |
| 232 |
% Clearly much comfort concerning \emph{routine} jobs is missing now, |
| 233 |
% 'txtproc' has more. Please sponsor (or otherwise support) the work for |
| 234 |
% improving this. |
| 235 |
% |
| 236 |
It should be noted (perhaps here) that the present approach to parsing |
| 237 |
is a quite \emph{simple} one and in this respect much different to the |
| 238 |
string handling mechanisms of |
| 239 |
'stringstrings',\footnote{\url{http://ctan.org/pkg/stringstrings}} |
| 240 |
'ted',\footnote{\url{http://ctan.org/pkg/ted}} |
| 241 |
'xstrings'\footnote{\url{http://ctan.org/pkg/xstrings}} |
| 242 |
(as I understand them, perhaps also |
| 243 |
'coolstr'\footnote{\url{http://ctan.org/pkg/coolstr}}) |
| 244 |
which are \emph{much more powerful} than what is offered here---but |
| 245 |
perhaps slow and for practical applications possibly replaceable by |
| 246 |
the present approach. |
| 247 |
\emph{Expandable replacement} seems not to exist outside 'fifinddo' |
| 248 |
(2009/04/13). |
| 249 |
|
| 250 |
Much is missing, I know.\footnote{There |
| 251 |
is more in my badly implemented 'txtproc.sty'.} |
| 252 |
I am just implementing what I actually need |
| 253 |
and what could show that this approach is worth being pursued. |
| 254 |
%% removed 2010/03/18: |
| 255 |
% It may need being sponsored or otherwise supported. |
| 256 |
|
| 257 |
\subsection{For insiders} |
| 258 |
|
| 259 |
\emph{Warning:} You may (at least at the present state of the work) |
| 260 |
have little success with this package, if you don't know about |
| 261 |
\TeX's category codes and how \TeX\ macros are defined. |
| 262 |
The package rather provides tools for package writers. |
| 263 |
You may, however, be able to run other packages which just load |
| 264 |
'fifinddo' as required background. |
| 265 |
|
| 266 |
That 'fifinddo' acts on ``\TeX(t)" files or so means that |
| 267 |
(at present) I think of applications on ``plain text" files which |
| 268 |
will usually be \TeX\ input files. ``At present" they are read |
| 269 |
without ``special characters," so essentially category codes of input |
| 270 |
characters are either 11 (``letter") or 12 (``other"). |
| 271 |
This way some things are easier than with usual \TeX\ applications: |
| 272 |
\begin{enumerate} |
| 273 |
\item You can ``look into" curly braces and ``behind" comment characters. |
| 274 |
\item There are exact or safe tests especially of \emph{empty macro arguments} |
| 275 |
that are ``expandable," i.e., they are ``robust," |
| 276 |
don't need assignments, can be executed in `\write'ing and in |
| 277 |
`\edef' definitions. ``Usually," the safe way to test emptiness |
| 278 |
is storing a macro argument as a macro, say `\tempo', in order |
| 279 |
to test `\ifx\tempo\empty' where `\empty' has been defined |
| 280 |
by `\def\empty{}' in the format. But this requires some |
| 281 |
`\def\tempo{#<n>}' which breaks in ``mere expanding" |
| 282 |
(\TeX\ \emph{evaluates} `\tempo' instead of defining it). |
| 283 |
An \emph{expandable} test on emptiness is, e.g. `\ifx$#<n>$', |
| 284 |
where we hope that it becomes `\iftrue' just if macro argument |
| 285 |
`#<n>' is empty indeed. However, ``usually" it may \emph{also} |
| 286 |
become `\iftrue' when `#<n>' starts with `$'---if the latter |
| 287 |
has category code 3 (``math shift"). But 'fifinddo' does |
| 288 |
not assign category code 3 to any character from the input file! |
| 289 |
Therefore `\ifx$#<n>$' is `\iftrue' \emph{exactly} if `#<n>' is |
| 290 |
empty. |
| 291 |
\item You can avoid interference with packages that are needed for |
| 292 |
typesetting. You can do the ``preprocessing" in one run with |
| 293 |
typesetting, but you should do the preprocessing before you |
| 294 |
load packages needed for typesetting. One may even try to keep |
| 295 |
the macros and settings for preprocessing local to a group. |
| 296 |
\end{enumerate} |
| 297 |
%% done, moving to .sty 2010/04/06: |
| 298 |
% Once there may be an option to read input with some usual \TeX\ |
| 299 |
% category codes as well, it may be useful to (some of) |
| 300 |
% \begin{itemize} |
| 301 |
% \item avoid matching substrings of control words, |
| 302 |
% \item skip blank spaces as \TeX\ does it usually, |
| 303 |
% \item catch \emph{balanced} input pieces, |
| 304 |
% \item ignore comments, |
| 305 |
% \item ignore certain characters. |
| 306 |
% \end{itemize} |
| 307 |
|
| 308 |
The essential approach of 'fifinddo' to looking for single strings is |
| 309 |
described in some detail in section~\ref{sec:theory}. |
| 310 |
|
| 311 |
The implementation of 'fifinddo' is as follows. |
| 312 |
User commands are specially highlighted (boxed\slash coloured), |
| 313 |
together with their syntax description. |
| 314 |
|
| 315 |
% \end{document} |
| 316 |
|
| 317 |
\section{Preliminaries} |
| 318 |
\subsection{Head of file (Legalese)} |
| 319 |
\sloppy |
| 320 |
\ProcessLineMessage{} |
| 321 |
\MakeInputJobDoc{23}{\SectionLevelTwoParseInput} |
| 322 |
\end{document} |
| 323 |
|
| 324 |
TODO cf. Kastrup 'makematch' and others 2009/04/09, |
| 325 |
'parselines' 2010/11/13 |
| 326 |
|
| 327 |
2009/04/10 title break, by -> with(!?); `boxed/coloured' |
| 328 |
2009/04/12 useful: own shorthands |
| 329 |
2009/04/13 substr.sty, \emph{expandable}, w/o `implement'!? |
| 330 |
2009/04/15 same with \in@ |
| 331 |
2010/03/08 EPS application removed |
| 332 |
2010/03/17 use \MakeInputJobDoc |
| 333 |
2010/03/18 `Missing' -> `Comparisons' etc.; \Require... |
| 334 |
2010/03/19 Catalogue \url; '' -> " |
| 335 |
2010/03/20 \ctanpkgref |
| 336 |
2010/03/22 debugging; done ... |
| 337 |
2010/03/23 more URLs; abstract "then shared"; |
| 338 |
removed dates from 2009, fdtxttex.tex/tpl |
| 339 |
2010/04/06 possible uses of reading TeX category codes moved to .sty |
| 340 |
2010/11/11 \thanks <- package version |
| 341 |
2010/11/12 package date updated, \author -> makedoc.cfg |
| 342 |
2010/11/13 package date updated; parselines |
| 343 |
2010/11/24 debugging |
| 344 |
2010/11/25f. using readprov.sty |
| 345 |
2010/11/27 reworked \ProvidesFile |
| 346 |
2011/08/22 PDF metadata; adjusting to makedoc.cfg |
| 347 |
2011/11/13 [fleqn] |
| 348 |
2011/11/19 \pagebreak |