Debian Codegraph @

.cargo/vendor/shlex/src/quoting_warning.md

raw
1 // vim: textwidth=99
2 /*
3 Meta note: This file is loaded as a .rs file by rustdoc only.
4 */
5 /*!
6
7 A more detailed version of the [warning at the top level](super#warning) about the `quote`/`join`
8 family of APIs.
9
10 In general, passing the output of these APIs to a shell should recover the original string(s).
11 This page lists cases where it fails to do so.
12
13 In noninteractive contexts, there are only minor issues. 'Noninteractive' includes shell scripts
14 and `sh -c` arguments, or even scripts `source`d from interactive shells. The issues are:
15
16 - [Nul bytes](#nul-bytes)
17
18 - [Overlong commands](#overlong-commands)
19
20 If you are writing directly to the stdin of an interactive (`-i`) shell (i.e., if you are
21 pretending to be a terminal), or if you are writing to a cooked-mode pty (even if the other end is
22 noninteractive), then there is a **severe** security issue:
23
24 - [Control characters](#control-characters-interactive-contexts-only)
25
26 Finally, there are some [solved issues](#solved-issues).
27
28 # List of issues
29
30 ## Nul bytes
31
32 For non-interactive shells, the most problematic input is nul bytes (bytes with value 0). The
33 non-deprecated functions all default to returning [`QuoteError::Nul`] when encountering them, but
34 the deprecated [`quote`] and [`join`] functions leave them as-is.
35
36 In Unix, nul bytes can't appear in command arguments, environment variables, or filenames. It's
37 not a question of proper quoting; they just can't be used at all. This is a consequence of Unix's
38 system calls all being designed around nul-terminated C strings.
39
40 Shells inherit that limitation. Most of them do not accept nul bytes in strings even internally.
41 Even when they do, it's pretty much useless or even dangerous, since you can't pass them to
42 external commands.
43
44 In some cases, you might fail to pass the nul byte to the shell in the first place. For example,
45 the following code uses [`join`] to tunnel a command over an SSH connection:
46
47 ```rust
48 std::process::Command::new("ssh")
49 .arg("myhost")
50 .arg("--")
51 .arg(join(my_cmd_args))
52 ```
53
54 If any argument in `my_cmd_args` contains a nul byte, then `join(my_cmd_args)` will contain a nul
55 byte. But `join(my_cmd_args)` is itself being passed as an argument to a command (the ssh
56 command), and command arguments can't contain nul bytes! So this will simply result in the
57 `Command` failing to launch.
58
59 Still, there are other ways to smuggle nul bytes into a shell. How the shell reacts depends on the
60 shell and the method of smuggling. For example, here is Bash 5.2.21 exhibiting three different
61 behaviors:
62
63 - With ANSI-C quoting, the string is truncated at the first nul byte:
64 ```bash
65 $ echo $'foo\0bar' | hexdump -C
66 00000000 66 6f 6f 0a |foo.|
67 ```
68
69 - With command substitution, nul bytes are removed with a warning:
70 ```bash
71 $ echo $(printf 'foo\0bar') | hexdump -C
72 bash: warning: command substitution: ignored null byte in input
73 00000000 66 6f 6f 62 61 72 0a |foobar.|
74 ```
75
76 - When a nul byte appears directly in a shell script, it's removed with no warning:
77 ```bash
78 $ printf 'echo "foo\0bar"' | bash | hexdump -C
79 00000000 66 6f 6f 62 61 72 0a |foobar.|
80 ```
81
82 Zsh, in contrast, actually allows nul bytes internally, in shell variables and even arguments to
83 builtin commands. But if a variable is exported to the environment, or if an argument is used for
84 an external command, then the child process will see it silently truncated at the first nul. This
85 might actually be more dangerous, depending on the use case.
86
87 ## Overlong commands
88
89 If you pass a long string into a shell, several things might happen:
90
91 - It might succeed, yet the shell might have trouble actually doing anything with it. For example:
92
93 ```bash
94 x=$(printf '%010000000d' 0); /bin/echo $x
95 bash: /bin/echo: Argument list too long
96 ```
97
98 - If you're using certain shells (e.g. Busybox Ash) *and* using a pty for communication, then the
99 shell will impose a line length limit, ignoring all input past the limit.
100
101 - If you're using a pty in cooked mode, then by default, if you write so many bytes as input that
102 it fills the kernel's internal buffer, the kernel will simply drop those bytes, instead of
103 blocking waiting for the shell to empty out the buffer. In other words, random bits of input can
104 be lost, which is obviously insecure.
105
106 Future versions of this crate may add an option to [`Quoter`] to check the length for you.
107
108 ## Control characters (*interactive contexts only*)
109
110 Control characters are the bytes from `\x00` to `\x1f`, plus `\x7f`. `\x00` (the nul byte) is
111 discussed [above](#nul-bytes), but what about the rest? Well, many of them correspond to terminal
112 keyboard shortcuts. For example, when you press Ctrl-A at a shell prompt, your terminal sends the
113 byte `\x01`. The shell sees that byte and (if not configured differently) takes the standard
114 action for Ctrl-A, which is to move the cursor to the beginning of the line.
115
116 This means that it's quite dangerous to pipe bytes to an interactive shell. For example, here is a
117 program that tries to tell Bash to echo an arbitrary string, 'safely':
118 ```rust
119 use std::process::{Command, Stdio};
120 use std::io::Write;
121
122 let evil_string = "\x01do_something_evil; ";
123 let quoted = shlex::try_quote(evil_string).unwrap();
124 println!("quoted string is {:?}", quoted);
125
126 let mut bash = Command::new("bash")
127 .arg("-i") // force interactive mode
128 .stdin(Stdio::piped())
129 .spawn()
130 .unwrap();
131 let stdin = bash.stdin.as_mut().unwrap();
132 write!(stdin, "echo {}\n", quoted).unwrap();
133 ```
134
135 Here's the output of the program (with irrelevant bits removed):
136
137 ```text
138 quoted string is "'\u{1}do_something_evil; '"
139 /tmp comex$ do_something_evil; 'echo '
140 bash: do_something_evil: command not found
141 bash: echo : command not found
142 ```
143
144 Even though we quoted it, Bash still ran an arbitrary command!
145
146 This is not because the quoting was insufficient, per se. In single quotes, all input is supposed
147 to be treated as raw data until the closing single quote. And in fact, this would work fine
148 without the `"-i"` argument.
149
150 But line input is a separate stage from shell syntax parsing. After all, if you type a single
151 quote on the keyboard, you wouldn't expect it to disable all your keyboard shortcuts. So a control
152 character always has its designated effect, no matter if it's quoted or backslash-escaped.
153
154 Also, some control characters are interpreted by the kernel tty layer instead, like CTRL-C to send
155 SIGINT. These can be an issue even with noninteractive shells, but only if using a pty for
156 communication, as opposed to a pipe.
157
158 To be safe, you just have to avoid sending them.
159
160 ### Why not just use hex escapes?
161
162 In any normal programming languages, this would be no big deal.
163
164 Any normal language has a way to escape arbitrary characters in strings by writing out their
165 numeric values. For example, Rust lets you write them in hexadecimal, like `"\x4f"` (or
166 `"\u{1d546}"` for Unicode). In this way, arbitrary strings can be represented using only 'nice'
167 simple characters. Any remotely suspicious character can be replaced with a numeric escape
168 sequence, where the escape sequence itself consists only of alphanumeric characters and some
169 punctuation. The result may not be the most readable[^choices], but it's quite safe from being
170 misinterpreted or corrupted in transit.
171
172 Shell is not normal. It has no numeric escape sequences.
173
174 There are a few different ways to quote characters (unquoted, unquoted-with-backslash, single
175 quotes, double quotes), but all of them involve writing the character itself. If the input
176 contains a control character, the output must contain that same character.
177
178 ### Mitigation: terminal filters
179
180 In practice, automating interactive shells like in the above example is pretty uncommon these days.
181 In most cases, the only way for a programmatically generated string to make its way to the input of
182 an interactive shell is if a human copies and pastes it into their terminal.
183
184 And many terminals detect when you paste a string containing control characters. iTerm2 strips
185 them out; gnome-terminal replaces them with alternate characters[^gr]; Kitty outright prompts for
186 confirmation. This mitigates the risk.
187
188 But it's not perfect. Some other terminals don't implement this check or implement it incorrectly.
189 Also, these checks tend to not filter the tab character, which could trigger tab completion. In
190 most cases that's a non-issue, because most shells support paste bracketing, which disables tab and
191 some other control characters[^bracketing] within pasted text. But in some cases paste bracketing
192 gets disabled.
193
194 ### Future possibility: ANSI-C quoting
195
196 I said that shell syntax has no numeric escapes, but that only applies to *portable* shell syntax.
197 Bash and Zsh support an obscure alternate quoting style with the syntax `$'foo'`. It's called
198 ["ANSI-C quoting"][ansic], and inside it you can use all the escape sequences supported by C,
199 including hex escapes:
200
201 ```bash
202 $ echo $'\x41\n\x42'
203 A
204 B
205 ```
206
207 But other shells don't support it — including Dash, a popular choice for `/bin/sh`, and Busybox's
208 Ash, frequently seen on stripped-down embedded systems. This crate's quoting functionality [tries
209 to be compatible](crate#compatibility) with those shells, plus all other POSIX-compatible shells.
210 That makes ANSI-C quoting a no-go.
211
212 Still, future versions of this crate may provide an option to enable ANSI-C quoting, at the cost of
213 reduced portability.
214
215 ### Future possibility: printf
216
217 Another option would be to invoke the `printf` command, which is required by POSIX to support octal
218 escapes. For example, you could 'escape' the Rust string `"\x01"` into the shell syntax `"$(printf
219 '\001')"`. The shell will execute the command `printf` with the first argument being literally a
220 backslash followed by three digits; `printf` will output the actual byte with value 1; and the
221 shell will substitute that back into the original command.
222
223 The problem is that 'escaping' a string into a command substitution just feels too surprising. If
224 nothing else, it only works with an actual shell; [other languages' shell parsing
225 routines](crate#compatibility) wouldn't understand it. Neither would this crate's own parser,
226 though that could be fixed.
227
228 Future versions of this crate may provide an option to use `printf` for quoting.
229
230 ### Special note: newlines
231
232 Did you know that `\r` and `\n` are control characters? They aren't as dangerous as other control
233 characters (if quoted properly). But there's still an issue with them in interactive contexts.
234
235 Namely, in some cases, interactive shells and/or the tty layer will 'helpfully' translate between
236 different line ending conventions. The possibilities include replacing `\r` with `\n`, replacing
237 `\n` with `\r\n`, and others. This can't result in command injection, but it's still a lossy
238 transformation which can result in a failure to round-trip (i.e. the shell sees a different string
239 from what was originally passed to `quote`).
240
241 Numeric escapes would solve this as well.
242
243 # Solved issues
244
245 ## Solved: Past vulnerability (GHSA-r7qv-8r2h-pg27 / RUSTSEC-2024-XXX)
246
247 Versions of this crate before 1.3.0 did not quote `{`, `}`, and `\xa0`.
248
249 See:
250 - <https://github.com/advisories/GHSA-r7qv-8r2h-pg27>
251 - (TODO: Add Rustsec link)
252
253 ## Solved: `!` and `^`
254
255 There are two non-control characters which have a special meaning in interactive contexts only: `!` and
256 `^`. Luckily, these can be escaped adequately.
257
258 The `!` character triggers [history expansion][he]; the `^` character can trigger a variant of
259 history expansion known as [Quick Substitution][qs]. Both of these characters get expanded even
260 inside of double-quoted strings\!
261
262 If we're in a double-quoted string, then we can't just escape these characters with a backslash.
263 Only a specific set of characters can be backslash-escaped inside double quotes; the set of
264 supported characters depends on the shell, but it often doesn't include `!` and `^`.[^escbs]
265 Trying to backslash-escape an unsupported character produces a literal backslash:
266 ```bash
267 $ echo "\!"
268 \!
269 ```
270
271 However, these characters don't get expanded in single-quoted strings, so this crate just
272 single-quotes them.
273
274 But there's a Bash bug where `^` actually does get partially expanded in single-quoted strings:
275 ```bash
276 $ echo '
277 > ^a^b
278 > '
279
280 !!:s^a^b
281 ```
282
283 To work around that, this crate forces `^` to appear right after an opening single quote. For
284 example, the string `"^` is quoted into `'"''^'` instead of `'"^'`. This restriction is overkill,
285 since `^` is only meaningful right after a newline, but it's a sufficient restriction (after all, a
286 `^` character can't be preceded by a newline if it's forced to be preceded by a single quote), and
287 for now it simplifies things.
288
289 ## Solved: `\xa0`
290
291 The byte `\xa0` may be treated as a shell word separator, specifically on Bash on macOS when using
292 the default UTF-8 locale, only when the input is invalid UTF-8. This crate handles the issue by
293 always using quotes for arguments containing this byte.
294
295 In fact, this crate always uses quotes for arguments containing any non-ASCII bytes. This may be
296 changed in the future, since it's a bit unfriendly to non-English users. But for now it
297 minimizes risk, especially considering the large number of different legacy single-byte locales
298 someone might hypothetically be running their shell in.
299
300 ### Demonstration
301
302 ```bash
303 $ echo -e 'ls a\xa0b' | bash
304 ls: a: No such file or directory
305 ls: b: No such file or directory
306 ```
307 The normal behavior would be to output a single line, e.g.:
308 ```bash
309 $ echo -e 'ls a\xa0b' | bash
310 ls: cannot access 'a'$'\240''b': No such file or directory
311 ```
312 (The specific quoting in the error doesn't matter.)
313
314 ### Cause
315
316 Just for fun, here's why this behavior occurs:
317
318 Bash decides which bytes serve as word separators based on the libc function [`isblank`][isblank].
319 On macOS on UTF-8 locales, this passes for `\xa0`, corresponding to U+00A0 NO-BREAK SPACE.
320
321 This is doubly unique compared to the other systems I tested (Linux/glibc, Linux/musl, and
322 Windows/MSVC). First, the other systems don't allow bytes in the range [0x80, 0xFF] to pass
323 <code>is<i>foo</i></code> functions in UTF-8 locales, even if the corresponding Unicode codepoint
324 does pass, as determined by the wide-character equivalent function, <code>isw<i>foo</i></code>.
325 Second, the other systems don't treat U+00A0 as blank (even using `iswblank`).
326
327 Meanwhile, Bash checks for multi-byte sequences and forbids them from being treated as special
328 characters, so the proper UTF-8 encoding of U+00A0, `b"\xc2\xa0"`, is not treated as a word
329 separator. Treatment as a word separator only happens for `b"\xa0"` alone, which is illegal UTF-8.
330
331 [ansic]: https://www.gnu.org/software/bash/manual/html_node/ANSI_002dC-Quoting.html
332 [he]: https://www.gnu.org/software/bash/manual/html_node/History-Interaction.html
333 [qs]: https://www.gnu.org/software/bash/manual/html_node/Event-Designators.html
334 [isblank]: https://man7.org/linux/man-pages/man3/isblank.3p.html
335 [nul]: #nul-bytes
336
337 [^choices]: This can lead to tough choices over which
338 characters to escape and which to leave as-is, especially when Unicode gets involved and you
339 have to balance the risk of confusion with the benefit of properly supporting non-English
340 languages.
341 <br>
342 <br>
343 We don't have the luxury of those choices.
344
345 [^gr]: For example, backspace (in Unicode lingo, U+0008 BACKSPACE) turns into U+2408 SYMBOL FOR BACKSPACE.
346
347 [^bracketing]: It typically disables almost all handling of control characters by the shell proper,
348 but one necessary exception is the end-of-paste sequence itself (which starts with the control
349 character `\x1b`). In addition, paste bracketing does not suppress handling of control
350 characters by the kernel tty layer, such as `\x03` sending SIGINT (which typically clears the
351 currently typed command, making it dangerous in a similar way to `\x01`).
352
353 [^escbs]: For example, Dash doesn't remove the backslash from `"\!"` because it simply doesn't know
354 anything about `!` as a special character: it doesn't support history expansion. On the other
355 end of the spectrum, Zsh supports history expansion and does remove the backslash — though only
356 in interactive mode. Bash's behavior is weirder. It supports history expansion, and if you
357 write `"\!"`, the backslash does prevent history expansion from occurring — but it doesn't get
358 removed!
359
360 */
361
362 // `use` declarations to make auto links work:
363 use ::{quote, join, Shlex, Quoter, QuoteError};
364
365 // TODO: add more about copy-paste and human readability.