Debian Codegraph @

Doc/library/shlex.rst

raw
1 :mod:`!shlex` --- Simple lexical analysis
2 =========================================
3
4 .. module:: shlex
5 :synopsis: Simple lexical analysis for Unix shell-like languages.
6
7 .. moduleauthor:: Eric S. Raymond <[email protected]>
8 .. moduleauthor:: Gustavo Niemeyer <[email protected]>
9 .. sectionauthor:: Eric S. Raymond <[email protected]>
10 .. sectionauthor:: Gustavo Niemeyer <[email protected]>
11
12 **Source code:** :source:`Lib/shlex.py`
13
14 --------------
15
16 The :class:`~shlex.shlex` class makes it easy to write lexical analyzers for
17 simple syntaxes resembling that of the Unix shell. This will often be useful
18 for writing minilanguages, (for example, in run control files for Python
19 applications) or for parsing quoted strings.
20
21 The :mod:`!shlex` module defines the following functions:
22
23
24 .. function:: split(s, comments=False, posix=True)
25
26 Split the string *s* using shell-like syntax. If *comments* is :const:`False`
27 (the default), the parsing of comments in the given string will be disabled
28 (setting the :attr:`~shlex.commenters` attribute of the
29 :class:`~shlex.shlex` instance to the empty string). This function operates
30 in POSIX mode by default, but uses non-POSIX mode if the *posix* argument is
31 false.
32
33 .. versionchanged:: 3.12
34 Passing ``None`` for *s* argument now raises an exception, rather than
35 reading :data:`sys.stdin`.
36
37 .. function:: join(split_command)
38
39 Concatenate the tokens of the list *split_command* and return a string.
40 This function is the inverse of :func:`split`.
41
42 >>> from shlex import join
43 >>> print(join(['echo', '-n', 'Multiple words']))
44 echo -n 'Multiple words'
45
46 The returned value is shell-escaped to protect against injection
47 vulnerabilities (see :func:`quote`).
48
49 .. versionadded:: 3.8
50
51
52 .. function:: quote(s)
53
54 Return a shell-escaped version of the string *s*. The returned value is a
55 string that can safely be used as one token in a shell command line, for
56 cases where you cannot use a list.
57
58 .. _shlex-quote-warning:
59
60 .. warning::
61
62 The ``shlex`` module is **only designed for Unix shells**.
63
64 The :func:`quote` function is not guaranteed to be correct on non-POSIX
65 compliant shells or shells from other operating systems such as Windows.
66 Executing commands quoted by this module on such shells can open up the
67 possibility of a command injection vulnerability.
68
69 Consider using functions that pass command arguments with lists such as
70 :func:`subprocess.run` with ``shell=False``.
71
72 This idiom would be unsafe:
73
74 >>> filename = 'somefile; rm -rf ~'
75 >>> command = 'ls -l {}'.format(filename)
76 >>> print(command) # executed by a shell: boom!
77 ls -l somefile; rm -rf ~
78
79 :func:`quote` lets you plug the security hole:
80
81 >>> from shlex import quote
82 >>> command = 'ls -l {}'.format(quote(filename))
83 >>> print(command)
84 ls -l 'somefile; rm -rf ~'
85 >>> remote_command = 'ssh home {}'.format(quote(command))
86 >>> print(remote_command)
87 ssh home 'ls -l '"'"'somefile; rm -rf ~'"'"''
88
89 The quoting is compatible with UNIX shells and with :func:`split`:
90
91 >>> from shlex import split
92 >>> remote_command = split(remote_command)
93 >>> remote_command
94 ['ssh', 'home', "ls -l 'somefile; rm -rf ~'"]
95 >>> command = split(remote_command[-1])
96 >>> command
97 ['ls', '-l', 'somefile; rm -rf ~']
98
99 .. versionadded:: 3.3
100
101 The :mod:`!shlex` module defines the following class:
102
103
104 .. class:: shlex(instream=None, infile=None, posix=False, punctuation_chars=False)
105
106 A :class:`~shlex.shlex` instance or subclass instance is a lexical analyzer
107 object. The initialization argument, if present, specifies where to read
108 characters from. It must be a file-/stream-like object with
109 :meth:`~io.TextIOBase.read` and :meth:`~io.TextIOBase.readline` methods, or
110 a string. If no argument is given, input will be taken from ``sys.stdin``.
111 The second optional argument is a filename string, which sets the initial
112 value of the :attr:`~shlex.infile` attribute. If the *instream*
113 argument is omitted or equal to ``sys.stdin``, this second argument
114 defaults to "stdin". The *posix* argument defines the operational mode:
115 when *posix* is not true (default), the :class:`~shlex.shlex` instance will
116 operate in compatibility mode. When operating in POSIX mode,
117 :class:`~shlex.shlex` will try to be as close as possible to the POSIX shell
118 parsing rules. The *punctuation_chars* argument provides a way to make the
119 behaviour even closer to how real shells parse. This can take a number of
120 values: the default value, ``False``, preserves the behaviour seen under
121 Python 3.5 and earlier. If set to ``True``, then parsing of the characters
122 ``();<>|&`` is changed: any run of these characters (considered punctuation
123 characters) is returned as a single token. If set to a non-empty string of
124 characters, those characters will be used as the punctuation characters. Any
125 characters in the :attr:`wordchars` attribute that appear in
126 *punctuation_chars* will be removed from :attr:`wordchars`. See
127 :ref:`improved-shell-compatibility` for more information. *punctuation_chars*
128 can be set only upon :class:`~shlex.shlex` instance creation and can't be
129 modified later.
130
131 .. versionchanged:: 3.6
132 The *punctuation_chars* parameter was added.
133
134 .. seealso::
135
136 Module :mod:`configparser`
137 Parser for configuration files similar to the Windows :file:`.ini` files.
138
139
140 .. _shlex-objects:
141
142 shlex Objects
143 -------------
144
145 A :class:`~shlex.shlex` instance has the following methods:
146
147
148 .. method:: shlex.get_token()
149
150 Return a token. If tokens have been stacked using :meth:`push_token`, pop a
151 token off the stack. Otherwise, read one from the input stream. If reading
152 encounters an immediate end-of-file, :attr:`eof` is returned (the empty
153 string (``''``) in non-POSIX mode, and ``None`` in POSIX mode).
154
155
156 .. method:: shlex.push_token(str)
157
158 Push the argument onto the token stack.
159
160
161 .. method:: shlex.read_token()
162
163 Read a raw token. Ignore the pushback stack, and do not interpret source
164 requests. (This is not ordinarily a useful entry point, and is documented here
165 only for the sake of completeness.)
166
167
168 .. method:: shlex.sourcehook(filename)
169
170 When :class:`~shlex.shlex` detects a source request (see :attr:`source`
171 below) this method is given the following token as argument, and expected
172 to return a tuple consisting of a filename and an open file-like object.
173
174 Normally, this method first strips any quotes off the argument. If the result
175 is an absolute pathname, or there was no previous source request in effect, or
176 the previous source was a stream (such as ``sys.stdin``), the result is left
177 alone. Otherwise, if the result is a relative pathname, the directory part of
178 the name of the file immediately before it on the source inclusion stack is
179 prepended (this behavior is like the way the C preprocessor handles ``#include
180 "file.h"``).
181
182 The result of the manipulations is treated as a filename, and returned as the
183 first component of the tuple, with :func:`open` called on it to yield the second
184 component. (Note: this is the reverse of the order of arguments in instance
185 initialization!)
186
187 This hook is exposed so that you can use it to implement directory search paths,
188 addition of file extensions, and other namespace hacks. There is no
189 corresponding 'close' hook, but a shlex instance will call the
190 :meth:`~io.IOBase.close` method of the sourced input stream when it returns
191 EOF.
192
193 For more explicit control of source stacking, use the :meth:`push_source` and
194 :meth:`pop_source` methods.
195
196
197 .. method:: shlex.push_source(newstream, newfile=None)
198
199 Push an input source stream onto the input stack. If the filename argument is
200 specified it will later be available for use in error messages. This is the
201 same method used internally by the :meth:`sourcehook` method.
202
203
204 .. method:: shlex.pop_source()
205
206 Pop the last-pushed input source from the input stack. This is the same method
207 used internally when the lexer reaches EOF on a stacked input stream.
208
209
210 .. method:: shlex.error_leader(infile=None, lineno=None)
211
212 This method generates an error message leader in the format of a Unix C compiler
213 error label; the format is ``'"%s", line %d: '``, where the ``%s`` is replaced
214 with the name of the current source file and the ``%d`` with the current input
215 line number (the optional arguments can be used to override these).
216
217 This convenience is provided to encourage :mod:`!shlex` users to generate error
218 messages in the standard, parseable format understood by Emacs and other Unix
219 tools.
220
221 Instances of :class:`~shlex.shlex` subclasses have some public instance
222 variables which either control lexical analysis or can be used for debugging:
223
224
225 .. attribute:: shlex.commenters
226
227 The string of characters that are recognized as comment beginners. All
228 characters from the comment beginner to end of line are ignored. Includes just
229 ``'#'`` by default.
230
231
232 .. attribute:: shlex.wordchars
233
234 The string of characters that will accumulate into multi-character tokens. By
235 default, includes all ASCII alphanumerics and underscore. In POSIX mode, the
236 accented characters in the Latin-1 set are also included. If
237 :attr:`punctuation_chars` is not empty, the characters ``~-./*?=``, which can
238 appear in filename specifications and command line parameters, will also be
239 included in this attribute, and any characters which appear in
240 ``punctuation_chars`` will be removed from ``wordchars`` if they are present
241 there. If :attr:`whitespace_split` is set to ``True``, this will have no
242 effect.
243
244
245 .. attribute:: shlex.whitespace
246
247 Characters that will be considered whitespace and skipped. Whitespace bounds
248 tokens. By default, includes space, tab, linefeed and carriage-return.
249
250
251 .. attribute:: shlex.escape
252
253 Characters that will be considered as escape. This will be only used in POSIX
254 mode, and includes just ``'\'`` by default.
255
256
257 .. attribute:: shlex.quotes
258
259 Characters that will be considered string quotes. The token accumulates until
260 the same quote is encountered again (thus, different quote types protect each
261 other as in the shell.) By default, includes ASCII single and double quotes.
262
263
264 .. attribute:: shlex.escapedquotes
265
266 Characters in :attr:`quotes` that will interpret escape characters defined in
267 :attr:`escape`. This is only used in POSIX mode, and includes just ``'"'`` by
268 default.
269
270
271 .. attribute:: shlex.whitespace_split
272
273 If ``True``, tokens will only be split in whitespaces. This is useful, for
274 example, for parsing command lines with :class:`~shlex.shlex`, getting
275 tokens in a similar way to shell arguments. When used in combination with
276 :attr:`punctuation_chars`, tokens will be split on whitespace in addition to
277 those characters.
278
279 .. versionchanged:: 3.8
280 The :attr:`punctuation_chars` attribute was made compatible with the
281 :attr:`whitespace_split` attribute.
282
283
284 .. attribute:: shlex.infile
285
286 The name of the current input file, as initially set at class instantiation time
287 or stacked by later source requests. It may be useful to examine this when
288 constructing error messages.
289
290
291 .. attribute:: shlex.instream
292
293 The input stream from which this :class:`~shlex.shlex` instance is reading
294 characters.
295
296
297 .. attribute:: shlex.source
298
299 This attribute is ``None`` by default. If you assign a string to it, that
300 string will be recognized as a lexical-level inclusion request similar to the
301 ``source`` keyword in various shells. That is, the immediately following token
302 will be opened as a filename and input will be taken from that stream until
303 EOF, at which point the :meth:`~io.IOBase.close` method of that stream will be
304 called and the input source will again become the original input stream. Source
305 requests may be stacked any number of levels deep.
306
307
308 .. attribute:: shlex.debug
309
310 If this attribute is numeric and ``1`` or more, a :class:`~shlex.shlex`
311 instance will print verbose progress output on its behavior. If you need
312 to use this, you can read the module source code to learn the details.
313
314
315 .. attribute:: shlex.lineno
316
317 Source line number (count of newlines seen so far plus one).
318
319
320 .. attribute:: shlex.token
321
322 The token buffer. It may be useful to examine this when catching exceptions.
323
324
325 .. attribute:: shlex.eof
326
327 Token used to determine end of file. This will be set to the empty string
328 (``''``), in non-POSIX mode, and to ``None`` in POSIX mode.
329
330
331 .. attribute:: shlex.punctuation_chars
332
333 A read-only property. Characters that will be considered punctuation. Runs of
334 punctuation characters will be returned as a single token. However, note that no
335 semantic validity checking will be performed: for example, '>>>' could be
336 returned as a token, even though it may not be recognised as such by shells.
337
338 .. versionadded:: 3.6
339
340
341 .. _shlex-parsing-rules:
342
343 Parsing Rules
344 -------------
345
346 When operating in non-POSIX mode, :class:`~shlex.shlex` will try to obey the
347 following rules.
348
349 * Quote characters are not recognized within words (``Do"Not"Separate`` is
350 parsed as the single word ``Do"Not"Separate``);
351
352 * Escape characters are not recognized;
353
354 * Enclosing characters in quotes preserve the literal value of all characters
355 within the quotes;
356
357 * Closing quotes separate words (``"Do"Separate`` is parsed as ``"Do"`` and
358 ``Separate``);
359
360 * If :attr:`~shlex.whitespace_split` is ``False``, any character not
361 declared to be a word character, whitespace, or a quote will be returned as
362 a single-character token. If it is ``True``, :class:`~shlex.shlex` will only
363 split words in whitespaces;
364
365 * EOF is signaled with an empty string (``''``);
366
367 * It's not possible to parse empty strings, even if quoted.
368
369 When operating in POSIX mode, :class:`~shlex.shlex` will try to obey the
370 following parsing rules.
371
372 * Quotes are stripped out, and do not separate words (``"Do"Not"Separate"`` is
373 parsed as the single word ``DoNotSeparate``);
374
375 * Non-quoted escape characters (e.g. ``'\'``) preserve the literal value of the
376 next character that follows;
377
378 * Enclosing characters in quotes which are not part of
379 :attr:`~shlex.escapedquotes` (e.g. ``"'"``) preserve the literal value
380 of all characters within the quotes;
381
382 * Enclosing characters in quotes which are part of
383 :attr:`~shlex.escapedquotes` (e.g. ``'"'``) preserves the literal value
384 of all characters within the quotes, with the exception of the characters
385 mentioned in :attr:`~shlex.escape`. The escape characters retain their
386 special meaning only when followed by the quote in use, or the escape
387 character itself. Otherwise the escape character will be considered a
388 normal character.
389
390 * EOF is signaled with a :const:`None` value;
391
392 * Quoted empty strings (``''``) are allowed.
393
394 .. _improved-shell-compatibility:
395
396 Improved Compatibility with Shells
397 ----------------------------------
398
399 .. versionadded:: 3.6
400
401 The :class:`shlex` class provides compatibility with the parsing performed by
402 common Unix shells like ``bash``, ``dash``, and ``sh``. To take advantage of
403 this compatibility, specify the ``punctuation_chars`` argument in the
404 constructor. This defaults to ``False``, which preserves pre-3.6 behaviour.
405 However, if it is set to ``True``, then parsing of the characters ``();<>|&``
406 is changed: any run of these characters is returned as a single token. While
407 this is short of a full parser for shells (which would be out of scope for the
408 standard library, given the multiplicity of shells out there), it does allow
409 you to perform processing of command lines more easily than you could
410 otherwise. To illustrate, you can see the difference in the following snippet:
411
412 .. doctest::
413 :options: +NORMALIZE_WHITESPACE
414
415 >>> import shlex
416 >>> text = "a && b; c && d || e; f >'abc'; (def \"ghi\")"
417 >>> s = shlex.shlex(text, posix=True)
418 >>> s.whitespace_split = True
419 >>> list(s)
420 ['a', '&&', 'b;', 'c', '&&', 'd', '||', 'e;', 'f', '>abc;', '(def', 'ghi)']
421 >>> s = shlex.shlex(text, posix=True, punctuation_chars=True)
422 >>> s.whitespace_split = True
423 >>> list(s)
424 ['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', 'abc', ';',
425 '(', 'def', 'ghi', ')']
426
427 Of course, tokens will be returned which are not valid for shells, and you'll
428 need to implement your own error checks on the returned tokens.
429
430 Instead of passing ``True`` as the value for the punctuation_chars parameter,
431 you can pass a string with specific characters, which will be used to determine
432 which characters constitute punctuation. For example::
433
434 >>> import shlex
435 >>> s = shlex.shlex("a && b || c", punctuation_chars="|")
436 >>> list(s)
437 ['a', '&', '&', 'b', '||', 'c']
438
439 .. note:: When ``punctuation_chars`` is specified, the :attr:`~shlex.wordchars`
440 attribute is augmented with the characters ``~-./*?=``. That is because these
441 characters can appear in file names (including wildcards) and command-line
442 arguments (e.g. ``--color=auto``). Hence::
443
444 >>> import shlex
445 >>> s = shlex.shlex('~/a && b-c --color=auto || d *.py?',
446 ... punctuation_chars=True)
447 >>> list(s)
448 ['~/a', '&&', 'b-c', '--color=auto', '||', 'd', '*.py?']
449
450 However, to match the shell as closely as possible, it is recommended to
451 always use ``posix`` and :attr:`~shlex.whitespace_split` when using
452 :attr:`~shlex.punctuation_chars`, which will negate
453 :attr:`~shlex.wordchars` entirely.
454
455 For best effect, ``punctuation_chars`` should be set in conjunction with
456 ``posix=True``. (Note that ``posix=False`` is the default for
457 :class:`~shlex.shlex`.)