Debian Codegraph @

Doc/library/asyncio-stream.rst

raw
1 .. currentmodule:: asyncio
2
3 .. _asyncio-streams:
4
5 =======
6 Streams
7 =======
8
9 **Source code:** :source:`Lib/asyncio/streams.py`
10
11 -------------------------------------------------
12
13 Streams are high-level async/await-ready primitives to work with
14 network connections. Streams allow sending and receiving data without
15 using callbacks or low-level protocols and transports.
16
17 .. _asyncio_example_stream:
18
19 Here is an example of a TCP echo client written using asyncio
20 streams::
21
22 import asyncio
23
24 async def tcp_echo_client(message):
25 reader, writer = await asyncio.open_connection(
26 '127.0.0.1', 8888)
27
28 print(f'Send: {message!r}')
29 writer.write(message.encode())
30 await writer.drain()
31
32 data = await reader.read(100)
33 print(f'Received: {data.decode()!r}')
34
35 print('Close the connection')
36 writer.close()
37 await writer.wait_closed()
38
39 asyncio.run(tcp_echo_client('Hello World!'))
40
41
42 See also the `Examples`_ section below.
43
44
45 .. rubric:: Stream Functions
46
47 The following top-level asyncio functions can be used to create
48 and work with streams:
49
50
51 .. function:: open_connection(host=None, port=None, *, \
52 limit=None, ssl=None, family=0, proto=0, \
53 flags=0, sock=None, local_addr=None, \
54 server_hostname=None, ssl_handshake_timeout=None, \
55 ssl_shutdown_timeout=None, \
56 happy_eyeballs_delay=None, interleave=None)
57 :async:
58
59 Establish a network connection and return a pair of
60 ``(reader, writer)`` objects.
61
62 The returned *reader* and *writer* objects are instances of
63 :class:`StreamReader` and :class:`StreamWriter` classes.
64
65 *limit* determines the buffer size limit used by the
66 returned :class:`StreamReader` instance. By default the *limit*
67 is set to 64 KiB.
68
69 The rest of the arguments are passed directly to
70 :meth:`loop.create_connection`.
71
72 .. note::
73
74 The *sock* argument transfers ownership of the socket to the
75 :class:`StreamWriter` created. To close the socket, call its
76 :meth:`~asyncio.StreamWriter.close` method.
77
78 .. versionchanged:: 3.7
79 Added the *ssl_handshake_timeout* parameter.
80
81 .. versionchanged:: 3.8
82 Added the *happy_eyeballs_delay* and *interleave* parameters.
83
84 .. versionchanged:: 3.10
85 Removed the *loop* parameter.
86
87 .. versionchanged:: 3.11
88 Added the *ssl_shutdown_timeout* parameter.
89
90
91 .. function:: start_server(client_connected_cb, host=None, \
92 port=None, *, limit=None, \
93 family=socket.AF_UNSPEC, \
94 flags=socket.AI_PASSIVE, sock=None, \
95 backlog=100, ssl=None, reuse_address=None, \
96 reuse_port=None, keep_alive=None, \
97 ssl_handshake_timeout=None, \
98 ssl_shutdown_timeout=None, start_serving=True)
99 :async:
100
101 Start a socket server.
102
103 The *client_connected_cb* callback is called whenever a new client
104 connection is established. It receives a ``(reader, writer)`` pair
105 as two arguments, instances of the :class:`StreamReader` and
106 :class:`StreamWriter` classes.
107
108 *client_connected_cb* can be a plain callable or a
109 :ref:`coroutine function <coroutine>`; if it is a coroutine function,
110 it will be automatically scheduled as a :class:`Task`.
111
112 *limit* determines the buffer size limit used by the
113 returned :class:`StreamReader` instance. By default the *limit*
114 is set to 64 KiB.
115
116 The rest of the arguments are passed directly to
117 :meth:`loop.create_server`.
118
119 .. note::
120
121 The *sock* argument transfers ownership of the socket to the
122 server created. To close the socket, call the server's
123 :meth:`~asyncio.Server.close` method.
124
125 .. versionchanged:: 3.7
126 Added the *ssl_handshake_timeout* and *start_serving* parameters.
127
128 .. versionchanged:: 3.10
129 Removed the *loop* parameter.
130
131 .. versionchanged:: 3.11
132 Added the *ssl_shutdown_timeout* parameter.
133
134 .. versionchanged:: 3.13
135 Added the *keep_alive* parameter.
136
137
138 .. rubric:: Unix Sockets
139
140 .. function:: open_unix_connection(path=None, *, limit=None, \
141 ssl=None, sock=None, server_hostname=None, \
142 ssl_handshake_timeout=None, ssl_shutdown_timeout=None)
143 :async:
144
145 Establish a Unix socket connection and return a pair of
146 ``(reader, writer)``.
147
148 Similar to :func:`open_connection` but operates on Unix sockets.
149
150 See also the documentation of :meth:`loop.create_unix_connection`.
151
152 .. note::
153
154 The *sock* argument transfers ownership of the socket to the
155 :class:`StreamWriter` created. To close the socket, call its
156 :meth:`~asyncio.StreamWriter.close` method.
157
158 .. availability:: Unix.
159
160 .. versionchanged:: 3.7
161 Added the *ssl_handshake_timeout* parameter.
162 The *path* parameter can now be a :term:`path-like object`
163
164 .. versionchanged:: 3.10
165 Removed the *loop* parameter.
166
167 .. versionchanged:: 3.11
168 Added the *ssl_shutdown_timeout* parameter.
169
170
171 .. function:: start_unix_server(client_connected_cb, path=None, \
172 *, limit=None, sock=None, backlog=100, ssl=None, \
173 ssl_handshake_timeout=None, \
174 ssl_shutdown_timeout=None, start_serving=True, cleanup_socket=True)
175 :async:
176
177 Start a Unix socket server.
178
179 Similar to :func:`start_server` but works with Unix sockets.
180
181 If *cleanup_socket* is true then the Unix socket will automatically
182 be removed from the filesystem when the server is closed, unless the
183 socket has been replaced after the server has been created.
184
185 See also the documentation of :meth:`loop.create_unix_server`.
186
187 .. note::
188
189 The *sock* argument transfers ownership of the socket to the
190 server created. To close the socket, call the server's
191 :meth:`~asyncio.Server.close` method.
192
193 .. availability:: Unix.
194
195 .. versionchanged:: 3.7
196 Added the *ssl_handshake_timeout* and *start_serving* parameters.
197 The *path* parameter can now be a :term:`path-like object`.
198
199 .. versionchanged:: 3.10
200 Removed the *loop* parameter.
201
202 .. versionchanged:: 3.11
203 Added the *ssl_shutdown_timeout* parameter.
204
205 .. versionchanged:: 3.13
206 Added the *cleanup_socket* parameter.
207
208
209 StreamReader
210 ============
211
212 .. class:: StreamReader
213
214 Represents a reader object that provides APIs to read data
215 from the IO stream. As an :term:`asynchronous iterable`, the
216 object supports the :keyword:`async for` statement.
217
218 It is not recommended to instantiate *StreamReader* objects
219 directly; use :func:`open_connection` and :func:`start_server`
220 instead.
221
222 .. method:: feed_eof()
223
224 Acknowledge the EOF.
225
226 .. method:: read(n=-1)
227 :async:
228
229 Read up to *n* bytes from the stream.
230
231 If *n* is not provided or set to ``-1``,
232 read until EOF, then return all read :class:`bytes`.
233 If EOF was received and the internal buffer is empty,
234 return an empty ``bytes`` object.
235
236 If *n* is ``0``, return an empty ``bytes`` object immediately.
237
238 If *n* is positive, return at most *n* available ``bytes``
239 as soon as at least 1 byte is available in the internal buffer.
240 If EOF is received before any byte is read, return an empty
241 ``bytes`` object.
242
243 .. method:: readline()
244 :async:
245
246 Read one line, where "line" is a sequence of bytes
247 ending with ``\n``.
248
249 If EOF is received and ``\n`` was not found, the method
250 returns partially read data.
251
252 If EOF is received and the internal buffer is empty,
253 return an empty ``bytes`` object.
254
255 .. method:: readexactly(n)
256 :async:
257
258 Read exactly *n* bytes.
259
260 Raise an :exc:`IncompleteReadError` if EOF is reached before *n*
261 can be read. Use the :attr:`IncompleteReadError.partial`
262 attribute to get the partially read data.
263
264 .. method:: readuntil(separator=b'\n')
265 :async:
266
267 Read data from the stream until *separator* is found.
268
269 On success, the data and separator will be removed from the
270 internal buffer (consumed). Returned data will include the
271 separator at the end.
272
273 If the amount of data read exceeds the configured stream limit, a
274 :exc:`LimitOverrunError` exception is raised, and the data
275 is left in the internal buffer and can be read again.
276
277 If EOF is reached before the complete separator is found,
278 an :exc:`IncompleteReadError` exception is raised, and the internal
279 buffer is reset. The :attr:`IncompleteReadError.partial` attribute
280 may contain a portion of the separator.
281
282 The *separator* may also be a tuple of separators. In this
283 case the return value will be the shortest possible that has any
284 separator as the suffix. For the purposes of :exc:`LimitOverrunError`,
285 the shortest possible separator is considered to be the one that
286 matched.
287
288 .. versionadded:: 3.5.2
289
290 .. versionchanged:: 3.13
291
292 The *separator* parameter may now be a :class:`tuple` of
293 separators.
294
295 .. method:: at_eof()
296
297 Return ``True`` if the buffer is empty and :meth:`feed_eof`
298 was called.
299
300
301 StreamWriter
302 ============
303
304 .. class:: StreamWriter
305
306 Represents a writer object that provides APIs to write data
307 to the IO stream.
308
309 It is not recommended to instantiate *StreamWriter* objects
310 directly; use :func:`open_connection` and :func:`start_server`
311 instead.
312
313 .. method:: write(data)
314
315 The method attempts to write the *data* to the underlying socket immediately.
316 If that fails, the data is queued in an internal write buffer until it can be
317 sent.
318
319 The *data* buffer should be a bytes, bytearray, or C-contiguous one-dimensional
320 memoryview object.
321
322 The method should be used along with the ``drain()`` method::
323
324 stream.write(data)
325 await stream.drain()
326
327
328 .. method:: writelines(data)
329
330 The method writes a list (or any iterable) of bytes to the underlying socket
331 immediately.
332 If that fails, the data is queued in an internal write buffer until it can be
333 sent.
334
335 The method should be used along with the ``drain()`` method::
336
337 stream.writelines(lines)
338 await stream.drain()
339
340 .. method:: close()
341
342 The method closes the stream and the underlying socket.
343
344 The method should be used, though not mandatory,
345 along with the ``wait_closed()`` method::
346
347 stream.close()
348 await stream.wait_closed()
349
350 .. method:: can_write_eof()
351
352 Return ``True`` if the underlying transport supports
353 the :meth:`write_eof` method, ``False`` otherwise.
354
355 .. method:: write_eof()
356
357 Close the write end of the stream after the buffered write
358 data is flushed.
359
360 .. attribute:: transport
361
362 Return the underlying asyncio transport.
363
364 .. method:: get_extra_info(name, default=None)
365
366 Access optional transport information; see
367 :meth:`BaseTransport.get_extra_info` for details.
368
369 .. method:: drain()
370 :async:
371
372 Wait until it is appropriate to resume writing to the stream.
373 Example::
374
375 writer.write(data)
376 await writer.drain()
377
378 This is a flow control method that interacts with the underlying
379 IO write buffer. When the size of the buffer reaches
380 the high watermark, *drain()* blocks until the size of the
381 buffer is drained down to the low watermark and writing can
382 be resumed. When there is nothing to wait for, the :meth:`drain`
383 returns immediately.
384
385 .. note::
386
387 When the write buffer is below the high watermark,
388 :meth:`drain` returns immediately without yielding to
389 the event loop. As a result, code which repeatedly calls
390 ``write()`` followed by ``await drain()`` may prevent other
391 tasks from running. To prevent blocking behavior, yield
392 to the event loop explicitly with ``await asyncio.sleep(0)``
393 (see :func:`asyncio.sleep`).
394
395 .. method:: start_tls(sslcontext, *, server_hostname=None, \
396 ssl_handshake_timeout=None, ssl_shutdown_timeout=None)
397 :async:
398
399 Upgrade an existing stream-based connection to TLS.
400
401 Parameters:
402
403 * *sslcontext*: a configured instance of :class:`~ssl.SSLContext`.
404
405 * *server_hostname*: sets or overrides the host name that the target
406 server's certificate will be matched against.
407
408 * *ssl_handshake_timeout* is the time in seconds to wait for the TLS
409 handshake to complete before aborting the connection. ``60.0`` seconds
410 if ``None`` (default).
411
412 * *ssl_shutdown_timeout* is the time in seconds to wait for the SSL shutdown
413 to complete before aborting the connection. ``30.0`` seconds if ``None``
414 (default).
415
416 .. versionadded:: 3.11
417
418 .. versionchanged:: 3.12
419 Added the *ssl_shutdown_timeout* parameter.
420
421
422 .. method:: is_closing()
423
424 Return ``True`` if the stream is closed or in the process of
425 being closed.
426
427 .. versionadded:: 3.7
428
429 .. method:: wait_closed()
430 :async:
431
432 Wait until the stream is closed.
433
434 Should be called after :meth:`close` to wait until the underlying
435 connection is closed, ensuring that all data has been flushed
436 before e.g. exiting the program.
437
438 .. versionadded:: 3.7
439
440
441 Examples
442 ========
443
444 .. _asyncio-tcp-echo-client-streams:
445
446 TCP echo client using streams
447 -----------------------------
448
449 TCP echo client using the :func:`asyncio.open_connection` function::
450
451 import asyncio
452
453 async def tcp_echo_client(message):
454 reader, writer = await asyncio.open_connection(
455 '127.0.0.1', 8888)
456
457 print(f'Send: {message!r}')
458 writer.write(message.encode())
459 await writer.drain()
460
461 data = await reader.read(100)
462 print(f'Received: {data.decode()!r}')
463
464 print('Close the connection')
465 writer.close()
466 await writer.wait_closed()
467
468 asyncio.run(tcp_echo_client('Hello World!'))
469
470
471 .. seealso::
472
473 The :ref:`TCP echo client protocol <asyncio_example_tcp_echo_client_protocol>`
474 example uses the low-level :meth:`loop.create_connection` method.
475
476
477 .. _asyncio-tcp-echo-server-streams:
478
479 TCP echo server using streams
480 -----------------------------
481
482 TCP echo server using the :func:`asyncio.start_server` function::
483
484 import asyncio
485
486 async def handle_echo(reader, writer):
487 data = await reader.read(100)
488 message = data.decode()
489 addr = writer.get_extra_info('peername')
490
491 print(f"Received {message!r} from {addr!r}")
492
493 print(f"Send: {message!r}")
494 writer.write(data)
495 await writer.drain()
496
497 print("Close the connection")
498 writer.close()
499 await writer.wait_closed()
500
501 async def main():
502 server = await asyncio.start_server(
503 handle_echo, '127.0.0.1', 8888)
504
505 addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
506 print(f'Serving on {addrs}')
507
508 async with server:
509 await server.serve_forever()
510
511 asyncio.run(main())
512
513
514 .. seealso::
515
516 The :ref:`TCP echo server protocol <asyncio_example_tcp_echo_server_protocol>`
517 example uses the :meth:`loop.create_server` method.
518
519
520 Get HTTP headers
521 ----------------
522
523 Simple example querying HTTP headers of the URL passed on the command line::
524
525 import asyncio
526 import urllib.parse
527 import sys
528
529 async def print_http_headers(url):
530 url = urllib.parse.urlsplit(url)
531 if url.scheme == 'https':
532 reader, writer = await asyncio.open_connection(
533 url.hostname, 443, ssl=True)
534 else:
535 reader, writer = await asyncio.open_connection(
536 url.hostname, 80)
537
538 query = (
539 f"HEAD {url.path or '/'} HTTP/1.0\r\n"
540 f"Host: {url.hostname}\r\n"
541 f"\r\n"
542 )
543
544 writer.write(query.encode('latin-1'))
545 while True:
546 line = await reader.readline()
547 if not line:
548 break
549
550 line = line.decode('latin1').rstrip()
551 if line:
552 print(f'HTTP header> {line}')
553
554 # Ignore the body, close the socket
555 writer.close()
556 await writer.wait_closed()
557
558 url = sys.argv[1]
559 asyncio.run(print_http_headers(url))
560
561
562 Usage::
563
564 python example.py http://example.com/path/page.html
565
566 or with HTTPS::
567
568 python example.py https://example.com/path/page.html
569
570
571 .. _asyncio_example_create_connection-streams:
572
573 Register an open socket to wait for data using streams
574 ------------------------------------------------------
575
576 Coroutine waiting until a socket receives data using the
577 :func:`open_connection` function::
578
579 import asyncio
580 import socket
581
582 async def wait_for_data():
583 # Get a reference to the current event loop because
584 # we want to access low-level APIs.
585 loop = asyncio.get_running_loop()
586
587 # Create a pair of connected sockets.
588 rsock, wsock = socket.socketpair()
589
590 # Register the open socket to wait for data.
591 reader, writer = await asyncio.open_connection(sock=rsock)
592
593 # Simulate the reception of data from the network
594 loop.call_soon(wsock.send, 'abc'.encode())
595
596 # Wait for data
597 data = await reader.read(100)
598
599 # Got data, we are done: close the socket
600 print("Received:", data.decode())
601 writer.close()
602 await writer.wait_closed()
603
604 # Close the second socket
605 wsock.close()
606
607 asyncio.run(wait_for_data())
608
609 .. seealso::
610
611 The :ref:`register an open socket to wait for data using a protocol
612 <asyncio_example_create_connection>` example uses a low-level protocol and
613 the :meth:`loop.create_connection` method.
614
615 The :ref:`watch a file descriptor for read events
616 <asyncio_example_watch_fd>` example uses the low-level
617 :meth:`loop.add_reader` method to watch a file descriptor.