Debian Codegraph @

texmf-dist/source/latex/download/download.tex

raw
1 %% download - download files with LaTeX
2 %%
3 %% Copyright (C) 2012-2019 by Simon Sigurdhsson <[email protected]>
4 %%
5 %% This work may be distributed and/or modified under the
6 %% conditions of the LaTeX Project Public License, either version 1.3
7 %% of this license or (at your option) any later version.
8 %% The latest version of this license is in
9 %% http://www.latex-project.org/lppl.txt
10 %% and version 1.3 or later is part of all distributions of LaTeX
11 %% version 2005/12/01 or later.
12 %%
13 %% This work has the LPPL maintenance status `maintained'.
14 %%
15 %% The Current Maintainer of this work is Simon Sigurdhsson.
16 %%
17 %% This work consists of the file download.tex
18 %% and the derived file download.sty.
19 \documentclass{skdoc}
20 \usepackage{hologo}
21 \usepackage[style=authoryear,backend=biber]{biblatex}
22 %%\usepackage{chslacite}
23
24 \ExplSyntaxOn
25 \cs_set_protected_nopar:Npn\ExplHack{
26 \char_set_catcode_letter:n{ 58 }
27 \char_set_catcode_letter:n{ 95 }
28 }
29 \ExplSyntaxOff
30 \ExplHack
31
32 \begin{filecontents}{download.bib}
33 @online{Klinger12,
34 author = {Max Klinger},
35 title = {Creating a URL downloading command to be used with e.g. \texttt{\textbackslash includegraphics}},
36 year = {2012},
37 url = {http://tex.stackexchange.com/questions/88430/creating-a-url-downloading-command-to-be-used-with-e-g-includegraphics}
38 }
39 \end{filecontents}
40 \addbibresource{download.bib}
41
42 \begin{document}
43 % Change & version info
44 \version{1.2}
45 \changes{1.0}{Initial version}
46 \changes{1.1}{Added aria2 and axel engines}
47 \changes{1.2}{Protect a bunch of macros and conditionals. Track \pkg{expl3} changes (thanks to Phelype Oleinik)}
48 % Don't forget to update the version number and release date of
49 % the package declaration in the implementation!
50
51 % Metadata
52 \package[ctan=download,vcs=https://github.com/urdh/download]{download}
53 \author{Simon Sigurdhsson}
54 \email{[email protected]}
55
56 % First page
57 \maketitle
58 \begin{abstract}
59 The \thepackage\ package allows \LaTeX\ to download files using
60 cURL, wget, aria2 or axel.
61 \end{abstract}
62
63 \section{Introduction}
64 This package, inspired by a question on \TeX.SE\footcite{Klinger12},
65 allows \LaTeX\ to download files using one of four engines.
66 Since it needs
67 to run external commands, it requires unrestricted \cs{write18}
68 access \Notice{do not indiscriminately run \hologo{pdfLaTeX} with
69 the \texttt{--shell-escape} flag; using this package it would be
70 possible to download malicious \texttt{.tex} that may abuse the
71 \cs{write18} access to harm your system}.
72
73 \section{Usage}
74 The package is very simple to use, but requires a *nix platform with
75 any of the engines installed and present in the \texttt{PATH}.
76
77 \subsection{Options}
78 \Option{engine}\WithValues{auto,curl,wget,aria2,axel}\AndDefault{auto}
79 The package only has one option, which controls what underlying
80 software is used to download the file. As of version \theversion,
81 the four engines available are cURL, wget, aria2 and axel.
82 The default, which is
83 used when no option is supplied, is \texttt{auto}. In this mode,
84 \thepkg\ will look for wget, cURL, aria2 and axel, in that order,
85 and use the first one available.
86
87 \subsection{Macros}
88 \DescribeMacro\download[<filename>]{<url>}
89 The only macro provided by \thepkg\ is \cs{download}. With it, you
90 can download any file from any \meta{url} supported by the underlying
91 engine (wget supports \texttt{http(s)} and \texttt{ftp}, cURL
92 supports a few more, aria2 supports torrent downloads and axel
93 supports downloading from multiple mirrors at once\footnote{See the
94 manpage of the respective command for more information.}; for most
95 cases wget should be enough). The
96 optional argument \meta{filename} makes the underlying engine save
97 the file with the specified filename \Notice{this also enables file
98 existence checking; without it, the engine will attempt to
99 download the file even if it exists --- wget and aria2 see the
100 existing file and do nothing, and axel probably replaces any existing
101 file but cURL will download a new copy with a numeral suffix}.
102
103 \Implementation
104 \SelfPreambleTo{\mypreamble}
105 \section{Implementation}
106 \DeclareFile[key=package,preamble=\mypreamble]{download.sty}
107 Let's have a look at the simple implementation. The package is based
108 on \LaTeX3, and should comply with the standards described i the
109 \pkg{expl3} manual. In any case, we begin by loading a few packages
110 (\pkg{expl3} for the \LaTeX3 core, \pkg{l3keys2e} for applying
111 \pkg*{l3keys} functionality to \LaTeXe\ package option parsing,
112 \pkg{pdftexcmds} for the \cs{pdf@shellescape} macro and \pkg{xparse}
113 for the public API definitions).
114 \begin{MacroCode}{package}
115 \RequirePackage{expl3,l3keys2e,pdftexcmds,xparse}
116 \end{MacroCode}
117
118 Then, we declare ourselves to provide the \thepkg\ \LaTeX3 package.
119 \begin{MacroCode}{package}
120 \ProvidesExplPackage{download}
121 {2019/10/01}{1.2}{download files with LaTeX}
122 \end{MacroCode}
123
124 \subsection{Messages}
125 We define a couple of messages using \pkg*{l3keys} functionality.
126
127 The two first messages will be used as fatal errors, when we notice
128 that functionality we absolutely \emph{require} (\emph{e.g.} either
129 unrestricted \cs{write18} or the specified engine) is missing.
130 \begin{MacroCode}{package}
131 \msg_new:nnnn{download}{no-write18}{Could~not~use~\string\write18!}
132 {Please~run~`latex`~with~the~`--shell-escape`~flag.}
133 \msg_new:nnnn{download}{no-engine}{Could~not~find~any~engine!}
134 {Please~make~sure~one~of~the~engines~is~installed~and~in~your~PATH.}
135 \end{MacroCode}
136
137 We also have a message being displayed when \cs{download} is being
138 used without its optional argument. This is a warning, since it may
139 imply that cURL is creating a lot of unwanted files.
140 \begin{MacroCode}{package}
141 \msg_new:nnnn{download}{no-name}{Using~\string\download\space~with~no~filename!}
142 {This~means~I~will~download~the~file~even~if~it~already~exists.}
143 \end{MacroCode}
144
145 The last two messages are diagnostics written to the log when
146 \opt{engine} is set to \texttt{auto}.
147 \begin{MacroCode}{package}
148 \msg_new:nnn{download}{use-curl}{Using~cURL.}
149 \msg_new:nnn{download}{use-wget}{Using~wget.}
150 \msg_new:nnn{download}{use-ariaII}{Using~aria2.}
151 \msg_new:nnn{download}{use-axel}{Using~axel.}
152 \end{MacroCode}
153
154 \subsection{The \cs{write18} test}
155 We require unrestricted \cs{write18} and as such we must test for
156 it. Using the \cs{pdf@shellescape} macro from \pkg{pdftexcmds}, we
157 can define a new conditional that decides if we have unrestricted
158 \cs{write18}.
159 \begin{macro}{\__download_if_shellescape:F}
160 \changes{1.2}{This conditional is now protected}
161 \begin{MacroCode}{package}
162 \prg_new_protected_conditional:Nnn\__download_if_shellescape:{F}{
163 \if_cs_exist:N\pdf@shellescape
164 \end{MacroCode}
165 If the command sequence exists (it really should), we test to see
166 if it is equal to one. The \cs{pdf@shellescape} macro will be zero
167 if no \cs{write18} access is available, two if we have restricted
168 access and one if access is unrestricted.
169 \begin{MacroCode}{package}
170 \if_int_compare:w\pdf@shellescape=\c_one_int
171 \prg_return_true:
172 \else:
173 \prg_return_false:
174 \fi:
175 \end{MacroCode}
176 If the command sequence doesn't exist, we assume that we have
177 unrestricted \cs{write18} access (we probably don't), and let
178 \LaTeX\ complain about it later.
179 \begin{MacroCode}{package}
180 \else:
181 \prg_return_true:
182 \fi:
183 }
184 \end{MacroCode}
185 \end{macro}
186
187 \subsection{Utility functions}
188 The existence tests for cURL and wget, explained later, create
189 a temporary file. We might as well clean that up at once, since
190 the user probably won't do that after each run of \LaTeX, and
191 an old file may break the check.
192 \begin{macro}{\__download_rm:n}[1]
193 {The file to be removed}
194 \changes{1.2}{This macro is now protected}
195 \begin{MacroCode}{package}
196 \cs_new_protected:Npn\__download_rm:n#1{
197 \immediate\write18{rm~#1}
198 }
199 \end{MacroCode}
200 \end{macro}
201
202 \subsection{Testing for the applications}
203 Testing for the existence of executables is done by calling
204 the standard *nix \texttt{which} command. We define one conditional
205 for all engines:
206 \begin{macro*}{\__download_if_executable_test:nF}
207 \begin{macro*}{\__download_if_executable_test:nT}
208 \begin{macro}{\__download_if_executable_test:nTF}[1]
209 {The executable to test the existence of}
210 \changes{1.1}{Condensed test macros into one macro with an argument}
211 \begin{MacroCode}{package}
212 \prg_new_conditional:Npnn\__download_if_executable_test:n#1{TF,T,F,p}{
213 \end{MacroCode}
214 First, run \texttt{which} to create the temporary file.
215 \begin{MacroCode}{package}
216 \immediate\write18{which~#1~&&~touch~\jobname.aex}
217 \end{MacroCode}
218 If the temporary file exists, we delete it and return true.
219 Otherwise, we return false.
220 \begin{MacroCode}{package}
221 \file_if_exist:nTF{\jobname.aex}{
222 \__download_rm:n{\jobname.aex}
223 \prg_return_true:
224 }{
225 \prg_return_false:
226 }
227 }
228 \end{MacroCode}
229 \end{macro}
230 \end{macro*}
231 \end{macro*}
232
233 \subsection{Using cURL and wget}
234 Actually using cURL and wget for downloading is simple, issuing
235 two different commands depending on wether we have the optional
236 argument or not (i.e. it is \cs{NoValue}).
237
238 \begin{macro}{\__download_curl_do:nn}[2]
239 {Filename to save file to, or \cs{NoValue}}
240 {URL to fetch the file from}
241 \changes{1.1}{cURL now follows redirects}
242 \changes{1.2}{This macro is now protected}
243 \begin{MacroCode}{package}
244 \cs_new_protected:Npn\__download_curl_do:nn#1#2{
245 \IfNoValueTF{#1}{
246 \end{MacroCode}
247 When no optional argument is given, we just invoke cURL with the
248 \texttt{-s} (silent) flag as well as the \texttt{-L} (follow
249 redirects) flag.
250 \begin{MacroCode}{package}
251 \immediate\write18{curl~-L~-s~#2}
252 }{
253 \end{MacroCode}
254 When we \emph{do} have an optional argument, we add the \texttt{-o}
255 flag to specify the output file.
256 \begin{MacroCode}{package}
257 \immediate\write18{curl~-L~-s~-o~#1~#2}
258 }
259 }
260 \end{MacroCode}
261 \end{macro}
262
263 \begin{macro}{\__download_wget_do:nn}[2]
264 {Filename to save file to, or \cs{NoValue}}
265 {URL to fetch the file from}
266 \changes{1.2}{This macro is now protected}
267 \begin{MacroCode}{package}
268 \cs_new_protected:Npn\__download_wget_do:nn#1#2{
269 \IfNoValueTF{#1}{
270 \end{MacroCode}
271 With wget, we pass the \texttt{-q} (quiet) flag as well as the
272 \texttt{-nc} (no clobber) flag, to avoid downloading files that
273 already exist.
274 \begin{MacroCode}{package}
275 \immediate\write18{wget~-q~-nc~#2}
276 }{
277 \end{MacroCode}
278 Again, when we have an optional argument we add the \texttt{-O}
279 flag to specify the output file.
280 \begin{MacroCode}{package}
281 \immediate\write18{wget~-q~-nc~-O~#1~#2}
282 }
283 }
284 \end{MacroCode}
285 \end{macro}
286
287 \begin{macro}{\__download_ariaII_do:nn}[2]
288 {Filename to save file to, or \cs{NoValue}}
289 {URL to fetch the file from}
290 \changes{1.2}{This macro is now protected}
291 \begin{MacroCode}{package}
292 \cs_new_protected:Npn\__download_ariaII_do:nn#1#2{
293 \IfNoValueTF{#1}{
294 \end{MacroCode}
295 With aria2, we pass the \texttt{-q} (quiet) flag as well as the
296 \texttt{--auto-file-renaming=false} (no clobber) flag, to avoid
297 creating a lot of duplicate files.
298 \begin{MacroCode}{package}
299 \immediate\write18{aria2c~-q~--auto-file-renaming=false~#2}
300 }{
301 \end{MacroCode}
302 Again, when we have an optional argument we add the \texttt{-o}
303 flag to specify the output file.
304 \begin{MacroCode}{package}
305 \immediate\write18{aria2c~-q~--auto-file-renaming=false~-o~#1~#2}
306 }
307 }
308 \end{MacroCode}
309 \end{macro}
310
311 \begin{macro}{\__download_axel_do:nn}[2]
312 {Filename to save file to, or \cs{NoValue}}
313 {URL to fetch the file from}
314 \changes{1.2}{This macro is now protected}
315 \begin{MacroCode}{package}
316 \cs_new_protected:Npn\__download_axel_do:nn#1#2{
317 \IfNoValueTF{#1}{
318 \end{MacroCode}
319 With axel, we pass the \texttt{-q} (quiet) flag.
320 \begin{MacroCode}{package}
321 \immediate\write18{axel~-q~#2}
322 }{
323 \end{MacroCode}
324 Again, when we have an optional argument we add the \texttt{-o}
325 flag to specify the output file.
326 \begin{MacroCode}{package}
327 \immediate\write18{axel~-q~-o~#1~#2}
328 }
329 }
330 \end{MacroCode}
331 \end{macro}
332
333 \subsection{The \texttt{auto} engine}
334 The automatic engine uses the tests and macros of the other engines
335 to provide functionality without selecting an engine. We first
336 define a conditional that, in essence, steps through the available
337 engines testing for their existence. If any of them exist, we're in
338 business.
339 \begin{macro*}{\__download_if_auto_test:TF}
340 \begin{macro}{\__download_if_auto_test:F}
341 \changes{1.1}{Added aria2c and axel engines to stack}
342 \changes{1.2}{This conditional is now protected}
343 \begin{MacroCode}{package}
344 \prg_new_protected_conditional:Nnn\__download_if_auto_test:{F,TF}{
345 \__download_if_executable_test:nTF{wget}{
346 \prg_return_true:
347 }{
348 \__download_if_executable_test:nTF{curl}{
349 \prg_return_true:
350 }{
351 \__download_if_executable_test:nTF{aria2c}{
352 \prg_return_true:
353 }{
354 \__download_if_executable_test:nTF{axel}{
355 \prg_return_true:
356 }{
357 \prg_return_false:
358 }
359 }
360 }
361 }
362 }
363 \end{MacroCode}
364 \end{macro}
365 \end{macro*}
366
367 We also define an automatic equivalent of the engine \texttt{_do}
368 macros, which selects the engines in the order wget, cURL, aria2
369 and axel.
370 \begin{macro}{\__download_auto_do:nn}[2]
371 {Filename to save file to, or \cs{NoValue}}
372 {URL to fetch the file from}
373 \changes{1.2}{This macro is now protected}
374 \begin{MacroCode}{package}
375 \cs_new_protected:Npn\__download_auto_do:nn#1#2{
376 \__download_if_executable_test:nTF{wget}{
377 \msg_info:nn{download}{use-wget}
378 \__download_wget_do:nn{#1}{#2}
379 }{
380 \__download_if_executable_test:nTF{curl}{
381 \msg_info:nn{download}{use-curl}
382 \__download_curl_do:nn{#1}{#2}
383 }{
384 \__download_if_executable_test:nTF{aria2c}{
385 \msg_info:nn{download}{use-ariaII}
386 \__download_ariaII_do:nn{#1}{#2}
387 }{
388 \msg_info:nn{download}{use-axel}
389 \__download_axel_do:nn{#1}{#2}
390 }
391 }
392 }
393 }
394 \end{MacroCode}
395 \end{macro}
396
397 \subsection{Package options}
398 As detailed earlier in the documentation, the only option of the
399 package is \opt{engine}. Here, we use the \pkg*{l3keys} functionality
400 to define a key-value system which we later use to read the package
401 options.
402 \begin{MacroCode}{package}
403 \keys_define:nn{download}{
404 engine .choice:,
405 \end{MacroCode}
406 \begin{macro}{\__download_do:nn}[2]
407 {Filename to save file to, or \cs{NoValue}}
408 {URL to fetch the file from}
409 \begin{macro}{\__download_if_auto_test:F}
410 First, the \texttt{auto} value. We globally define two macros as
411 aliases to the underlying \texttt{_do} and \texttt{_if_test} macros.
412 \begin{MacroCode}{package}
413 engine / auto .code:n =
414 {\cs_gset_eq:NN\__download_do:nn\__download_auto_do:nn
415 \prg_set_conditional:Nnn\__download_if_test:{F}{
416 \__download_if_auto_test:TF
417 {\prg_return_true:}{\prg_return_false:}}},
418 \end{MacroCode}
419 We do the same for the other options.
420 \begin{MacroCode}{package}
421 engine / curl .code:n =
422 {\cs_gset_eq:NN\__download_do:nn\__download_curl_do:nn
423 \prg_set_conditional:Nnn\__download_if_test:{F}{
424 \__download_if_executable_test:nTF{curl}
425 {\prg_return_true:}{\prg_return_false:}}},
426 engine / wget .code:n =
427 {\cs_gset_eq:NN\__download_do:nn\__download_wget_do:nn
428 \prg_set_conditional:Nnn\__download_if_test:{F}{
429 \__download_if_executable_test:nTF{wget}
430 {\prg_return_true:}{\prg_return_false:}}},
431 engine / aria2 .code:n =
432 {\cs_gset_eq:NN\__download_do:nn\__download_ariaII_do:nn
433 \prg_set_conditional:Nnn\__download_if_test:{F}{
434 \__download_if_executable_test:nTF{aria2c}
435 {\prg_return_true:}{\prg_return_false:}}},
436 engine / axel .code:n =
437 {\cs_gset_eq:NN\__download_do:nn\__download_axel_do:nn
438 \prg_set_conditional:Nnn\__download_if_test:{F}{
439 \__download_if_executable_test:nTF{axel}
440 {\prg_return_true:}{\prg_return_false:}}},
441 \end{MacroCode}
442 \end{macro}
443 \end{macro}
444 Lastly, we initialize the option with the \texttt{auto} value.
445 \begin{MacroCode}{package}
446 engine .initial:n = auto,
447 engine .default:n = auto,
448 }
449 \end{MacroCode}
450 Now, given the key-value system, we parse the options.
451 \begin{MacroCode}{package}
452 \ProcessKeysPackageOptions{download}
453 \end{MacroCode}
454
455 \subsection{Performing the tests}
456 Now that we know what engine we will be using, we can check for
457 \cs{write18} support and engine existence.
458 \begin{MacroCode}{package}
459 \__download_if_shellescape:F{\msg_fatal:nn{download}{no-write18}}
460 \__download_if_test:F{\msg_fatal:nn{download}{no-engine}}
461 \end{MacroCode}
462
463 \subsection{Public API}
464 The public API consists of only one macro, \cs{download}. It
465 simply calls the backend macro, unless the optional argument
466 is given and the file exists. If the file doesn't exist, it
467 also emits a friendly warning.
468 \begin{macro}{\download}[2]
469 {Filename to save file to, or \cs{NoValue}}
470 {URL to fetch the file from}
471 \begin{MacroCode}{package}
472 \DeclareDocumentCommand\download{om}{
473 \IfNoValueTF{#1}{
474 \msg_warning:nn{download}{no-name}
475 \__download_do:nn{#1}{#2}
476 }{
477 \file_if_exist:nTF{#1}{}{\__download_do:nn{#1}{#2}}
478 }
479 }
480 \end{MacroCode}
481 \end{macro}
482
483 And we're done.
484 \begin{MacroCode}{package}
485 \endinput
486 \end{MacroCode}
487
488 \Finale
489 \section{Installation}
490 The easiest way to install this package is using the package
491 manager provided by your \LaTeX\ installation if such a program
492 is available. Failing that, provided you have obtained the package
493 source (\file{download.tex} and \file{Makefile}) from either CTAN
494 or Github, running \texttt{make install} inside the source directory
495 works well. This will extract the documentation and code from
496 \file{download.tex}, install all files into the TDS tree at
497 \texttt{TEXMFHOME} and run \texttt{mktexlsr}.
498
499 If you want to extract code and documentation without installing
500 the package, run \texttt{make all} instead. If you insist on not
501 using \texttt{make}, remember that packages distributed using
502 \pkg{skdoc} must be extracted using \texttt{pdflatex}, \emph{not}
503 \texttt{tex} or \texttt{latex}.
504
505 \PrintChanges
506 \PrintIndex
507 \printbibliography
508 \end{document}