Debian Codegraph @

Doc/library/asyncio-stream.rst

raw
1 .. currentmodule:: asyncio
2
3 .. _asyncio-streams:
4
5 =======
6 Streams
7 =======
8
9 **Source code:** :source:`Lib/asyncio/streams.py`
10
11 -------------------------------------------------
12
13 Streams are high-level async/await-ready primitives to work with
14 network connections. Streams allow sending and receiving data without
15 using callbacks or low-level protocols and transports.
16
17 .. _asyncio_example_stream:
18
19 Here is an example of a TCP echo client written using asyncio
20 streams::
21
22 import asyncio
23
24 async def tcp_echo_client(message):
25 reader, writer = await asyncio.open_connection(
26 '127.0.0.1', 8888)
27
28 print(f'Send: {message!r}')
29 writer.write(message.encode())
30 await writer.drain()
31
32 data = await reader.read(100)
33 print(f'Received: {data.decode()!r}')
34
35 print('Close the connection')
36 writer.close()
37 await writer.wait_closed()
38
39 asyncio.run(tcp_echo_client('Hello World!'))
40
41
42 See also the `Examples`_ section below.
43
44
45 .. rubric:: Stream Functions
46
47 The following top-level asyncio functions can be used to create
48 and work with streams:
49
50
51 .. function:: open_connection(host=None, port=None, *, \
52 limit=None, ssl=None, family=0, proto=0, \
53 flags=0, sock=None, local_addr=None, \
54 server_hostname=None, ssl_handshake_timeout=None, \
55 ssl_shutdown_timeout=None, \
56 happy_eyeballs_delay=None, interleave=None)
57 :async:
58
59 Establish a network connection and return a pair of
60 ``(reader, writer)`` objects.
61
62 The returned *reader* and *writer* objects are instances of
63 :class:`StreamReader` and :class:`StreamWriter` classes.
64
65 *limit* determines the buffer size limit used by the
66 returned :class:`StreamReader` instance. By default the *limit*
67 is set to 64 KiB.
68
69 The rest of the arguments are passed directly to
70 :meth:`loop.create_connection`.
71
72 .. note::
73
74 The *sock* argument transfers ownership of the socket to the
75 :class:`StreamWriter` created. To close the socket, call its
76 :meth:`~asyncio.StreamWriter.close` method.
77
78 .. versionchanged:: 3.7
79 Added the *ssl_handshake_timeout* parameter.
80
81 .. versionchanged:: 3.8
82 Added the *happy_eyeballs_delay* and *interleave* parameters.
83
84 .. versionchanged:: 3.10
85 Removed the *loop* parameter.
86
87 .. versionchanged:: 3.11
88 Added the *ssl_shutdown_timeout* parameter.
89
90
91 .. function:: start_server(client_connected_cb, host=None, \
92 port=None, *, limit=None, \
93 family=socket.AF_UNSPEC, \
94 flags=socket.AI_PASSIVE, sock=None, \
95 backlog=100, ssl=None, reuse_address=None, \
96 reuse_port=None, keep_alive=None, \
97 ssl_handshake_timeout=None, \
98 ssl_shutdown_timeout=None, start_serving=True)
99 :async:
100
101 Start a socket server.
102
103 The *client_connected_cb* callback is called whenever a new client
104 connection is established. It receives a ``(reader, writer)`` pair
105 as two arguments, instances of the :class:`StreamReader` and
106 :class:`StreamWriter` classes.
107
108 *client_connected_cb* can be a plain callable or a
109 :ref:`coroutine function <coroutine>`; if it is a coroutine function,
110 it will be automatically scheduled as a :class:`Task`.
111
112 *limit* determines the buffer size limit used by the
113 returned :class:`StreamReader` instance. By default the *limit*
114 is set to 64 KiB.
115
116 The rest of the arguments are passed directly to
117 :meth:`loop.create_server`.
118
119 .. note::
120
121 The *sock* argument transfers ownership of the socket to the
122 server created. To close the socket, call the server's
123 :meth:`~asyncio.Server.close` method.
124
125 .. versionchanged:: 3.7
126 Added the *ssl_handshake_timeout* and *start_serving* parameters.
127
128 .. versionchanged:: 3.10
129 Removed the *loop* parameter.
130
131 .. versionchanged:: 3.11
132 Added the *ssl_shutdown_timeout* parameter.
133
134 .. versionchanged:: 3.13
135 Added the *keep_alive* parameter.
136
137
138 .. rubric:: Unix Sockets
139
140 .. function:: open_unix_connection(path=None, *, limit=None, \
141 ssl=None, sock=None, server_hostname=None, \
142 ssl_handshake_timeout=None, ssl_shutdown_timeout=None)
143 :async:
144
145 Establish a Unix socket connection and return a pair of
146 ``(reader, writer)``.
147
148 Similar to :func:`open_connection` but operates on Unix sockets.
149
150 See also the documentation of :meth:`loop.create_unix_connection`.
151
152 .. note::
153
154 The *sock* argument transfers ownership of the socket to the
155 :class:`StreamWriter` created. To close the socket, call its
156 :meth:`~asyncio.StreamWriter.close` method.
157
158 .. availability:: Unix.
159
160 .. versionchanged:: 3.7
161 Added the *ssl_handshake_timeout* parameter.
162 The *path* parameter can now be a :term:`path-like object`
163
164 .. versionchanged:: 3.10
165 Removed the *loop* parameter.
166
167 .. versionchanged:: 3.11
168 Added the *ssl_shutdown_timeout* parameter.
169
170
171 .. function:: start_unix_server(client_connected_cb, path=None, \
172 *, limit=None, sock=None, backlog=100, ssl=None, \
173 ssl_handshake_timeout=None, \
174 ssl_shutdown_timeout=None, start_serving=True, cleanup_socket=True)
175 :async:
176
177 Start a Unix socket server.
178
179 Similar to :func:`start_server` but works with Unix sockets.
180
181 If *cleanup_socket* is true then the Unix socket will automatically
182 be removed from the filesystem when the server is closed, unless the
183 socket has been replaced after the server has been created.
184
185 See also the documentation of :meth:`loop.create_unix_server`.
186
187 .. note::
188
189 The *sock* argument transfers ownership of the socket to the
190 server created. To close the socket, call the server's
191 :meth:`~asyncio.Server.close` method.
192
193 .. availability:: Unix.
194
195 .. versionchanged:: 3.7
196 Added the *ssl_handshake_timeout* and *start_serving* parameters.
197 The *path* parameter can now be a :term:`path-like object`.
198
199 .. versionchanged:: 3.10
200 Removed the *loop* parameter.
201
202 .. versionchanged:: 3.11
203 Added the *ssl_shutdown_timeout* parameter.
204
205 .. versionchanged:: 3.13
206 Added the *cleanup_socket* parameter.
207
208
209 StreamReader
210 ============
211
212 .. class:: StreamReader
213
214 Represents a reader object that provides APIs to read data
215 from the IO stream. As an :term:`asynchronous iterable`, the
216 object supports the :keyword:`async for` statement.
217
218 It is not recommended to instantiate *StreamReader* objects
219 directly; use :func:`open_connection` and :func:`start_server`
220 instead.
221
222 .. method:: feed_eof()
223
224 Acknowledge the EOF.
225
226 .. method:: read(n=-1)
227 :async:
228
229 Read up to *n* bytes from the stream.
230
231 If *n* is not provided or set to ``-1``,
232 read until EOF, then return all read :class:`bytes`.
233 If EOF was received and the internal buffer is empty,
234 return an empty ``bytes`` object.
235
236 If *n* is ``0``, return an empty ``bytes`` object immediately.
237
238 If *n* is positive, return at most *n* available ``bytes``
239 as soon as at least 1 byte is available in the internal buffer.
240 If EOF is received before any byte is read, return an empty
241 ``bytes`` object.
242
243 .. method:: readline()
244 :async:
245
246 Read one line, where "line" is a sequence of bytes
247 ending with ``\n``.
248
249 If EOF is received and ``\n`` was not found, the method
250 returns partially read data.
251
252 If EOF is received and the internal buffer is empty,
253 return an empty ``bytes`` object.
254
255 .. method:: readexactly(n)
256 :async:
257
258 Read exactly *n* bytes.
259
260 Raise an :exc:`IncompleteReadError` if EOF is reached before *n*
261 can be read. Use the :attr:`IncompleteReadError.partial`
262 attribute to get the partially read data.
263
264 .. method:: readuntil(separator=b'\n')
265 :async:
266
267 Read data from the stream until *separator* is found.
268
269 On success, the data and separator will be removed from the
270 internal buffer (consumed). Returned data will include the
271 separator at the end.
272
273 If the amount of data read exceeds the configured stream limit, a
274 :exc:`LimitOverrunError` exception is raised, and the data
275 is left in the internal buffer and can be read again.
276
277 If EOF is reached before the complete separator is found,
278 an :exc:`IncompleteReadError` exception is raised, and the internal
279 buffer is reset. The :attr:`IncompleteReadError.partial` attribute
280 may contain a portion of the separator.
281
282 The *separator* may also be a tuple of separators. In this
283 case the return value will be the shortest possible that has any
284 separator as the suffix. For the purposes of :exc:`LimitOverrunError`,
285 the shortest possible separator is considered to be the one that
286 matched.
287
288 .. versionadded:: 3.5.2
289
290 .. versionchanged:: 3.13
291
292 The *separator* parameter may now be a :class:`tuple` of
293 separators.
294
295 .. method:: at_eof()
296
297 Return ``True`` if the buffer is empty and :meth:`feed_eof`
298 was called.
299
300
301 StreamWriter
302 ============
303
304 .. class:: StreamWriter
305
306 Represents a writer object that provides APIs to write data
307 to the IO stream.
308
309 It is not recommended to instantiate *StreamWriter* objects
310 directly; use :func:`open_connection` and :func:`start_server`
311 instead.
312
313 .. method:: write(data)
314
315 The method attempts to write the *data* to the underlying socket immediately.
316 If that fails, the data is queued in an internal write buffer until it can be
317 sent.
318
319 The *data* buffer should be a bytes, bytearray, or C-contiguous one-dimensional
320 memoryview object.
321
322 The method should be used along with the ``drain()`` method::
323
324 stream.write(data)
325 await stream.drain()
326
327
328 .. method:: writelines(data)
329
330 The method writes a list (or any iterable) of bytes to the underlying socket
331 immediately.
332 If that fails, the data is queued in an internal write buffer until it can be
333 sent.
334
335 The method should be used along with the ``drain()`` method::
336
337 stream.writelines(lines)
338 await stream.drain()
339
340 .. method:: close()
341
342 The method closes the stream and the underlying socket.
343
344 The method should be used, though not mandatory,
345 along with the ``wait_closed()`` method::
346
347 stream.close()
348 await stream.wait_closed()
349
350 .. method:: can_write_eof()
351
352 Return ``True`` if the underlying transport supports
353 the :meth:`write_eof` method, ``False`` otherwise.
354
355 .. method:: write_eof()
356
357 Close the write end of the stream after the buffered write
358 data is flushed.
359
360 .. attribute:: transport
361
362 Return the underlying asyncio transport.
363
364 .. method:: get_extra_info(name, default=None)
365
366 Access optional transport information; see
367 :meth:`BaseTransport.get_extra_info` for details.
368
369 .. method:: drain()
370 :async:
371
372 Wait until it is appropriate to resume writing to the stream.
373 Example::
374
375 writer.write(data)
376 await writer.drain()
377
378 This is a flow control method that interacts with the underlying
379 IO write buffer. When the size of the buffer reaches
380 the high watermark, *drain()* blocks until the size of the
381 buffer is drained down to the low watermark and writing can
382 be resumed. When there is nothing to wait for, the :meth:`drain`
383 returns immediately.
384
385 .. method:: start_tls(sslcontext, *, server_hostname=None, \
386 ssl_handshake_timeout=None, ssl_shutdown_timeout=None)
387 :async:
388
389 Upgrade an existing stream-based connection to TLS.
390
391 Parameters:
392
393 * *sslcontext*: a configured instance of :class:`~ssl.SSLContext`.
394
395 * *server_hostname*: sets or overrides the host name that the target
396 server's certificate will be matched against.
397
398 * *ssl_handshake_timeout* is the time in seconds to wait for the TLS
399 handshake to complete before aborting the connection. ``60.0`` seconds
400 if ``None`` (default).
401
402 * *ssl_shutdown_timeout* is the time in seconds to wait for the SSL shutdown
403 to complete before aborting the connection. ``30.0`` seconds if ``None``
404 (default).
405
406 .. versionadded:: 3.11
407
408 .. versionchanged:: 3.12
409 Added the *ssl_shutdown_timeout* parameter.
410
411
412 .. method:: is_closing()
413
414 Return ``True`` if the stream is closed or in the process of
415 being closed.
416
417 .. versionadded:: 3.7
418
419 .. method:: wait_closed()
420 :async:
421
422 Wait until the stream is closed.
423
424 Should be called after :meth:`close` to wait until the underlying
425 connection is closed, ensuring that all data has been flushed
426 before e.g. exiting the program.
427
428 .. versionadded:: 3.7
429
430
431 Examples
432 ========
433
434 .. _asyncio-tcp-echo-client-streams:
435
436 TCP echo client using streams
437 -----------------------------
438
439 TCP echo client using the :func:`asyncio.open_connection` function::
440
441 import asyncio
442
443 async def tcp_echo_client(message):
444 reader, writer = await asyncio.open_connection(
445 '127.0.0.1', 8888)
446
447 print(f'Send: {message!r}')
448 writer.write(message.encode())
449 await writer.drain()
450
451 data = await reader.read(100)
452 print(f'Received: {data.decode()!r}')
453
454 print('Close the connection')
455 writer.close()
456 await writer.wait_closed()
457
458 asyncio.run(tcp_echo_client('Hello World!'))
459
460
461 .. seealso::
462
463 The :ref:`TCP echo client protocol <asyncio_example_tcp_echo_client_protocol>`
464 example uses the low-level :meth:`loop.create_connection` method.
465
466
467 .. _asyncio-tcp-echo-server-streams:
468
469 TCP echo server using streams
470 -----------------------------
471
472 TCP echo server using the :func:`asyncio.start_server` function::
473
474 import asyncio
475
476 async def handle_echo(reader, writer):
477 data = await reader.read(100)
478 message = data.decode()
479 addr = writer.get_extra_info('peername')
480
481 print(f"Received {message!r} from {addr!r}")
482
483 print(f"Send: {message!r}")
484 writer.write(data)
485 await writer.drain()
486
487 print("Close the connection")
488 writer.close()
489 await writer.wait_closed()
490
491 async def main():
492 server = await asyncio.start_server(
493 handle_echo, '127.0.0.1', 8888)
494
495 addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
496 print(f'Serving on {addrs}')
497
498 async with server:
499 await server.serve_forever()
500
501 asyncio.run(main())
502
503
504 .. seealso::
505
506 The :ref:`TCP echo server protocol <asyncio_example_tcp_echo_server_protocol>`
507 example uses the :meth:`loop.create_server` method.
508
509
510 Get HTTP headers
511 ----------------
512
513 Simple example querying HTTP headers of the URL passed on the command line::
514
515 import asyncio
516 import urllib.parse
517 import sys
518
519 async def print_http_headers(url):
520 url = urllib.parse.urlsplit(url)
521 if url.scheme == 'https':
522 reader, writer = await asyncio.open_connection(
523 url.hostname, 443, ssl=True)
524 else:
525 reader, writer = await asyncio.open_connection(
526 url.hostname, 80)
527
528 query = (
529 f"HEAD {url.path or '/'} HTTP/1.0\r\n"
530 f"Host: {url.hostname}\r\n"
531 f"\r\n"
532 )
533
534 writer.write(query.encode('latin-1'))
535 while True:
536 line = await reader.readline()
537 if not line:
538 break
539
540 line = line.decode('latin1').rstrip()
541 if line:
542 print(f'HTTP header> {line}')
543
544 # Ignore the body, close the socket
545 writer.close()
546 await writer.wait_closed()
547
548 url = sys.argv[1]
549 asyncio.run(print_http_headers(url))
550
551
552 Usage::
553
554 python example.py http://example.com/path/page.html
555
556 or with HTTPS::
557
558 python example.py https://example.com/path/page.html
559
560
561 .. _asyncio_example_create_connection-streams:
562
563 Register an open socket to wait for data using streams
564 ------------------------------------------------------
565
566 Coroutine waiting until a socket receives data using the
567 :func:`open_connection` function::
568
569 import asyncio
570 import socket
571
572 async def wait_for_data():
573 # Get a reference to the current event loop because
574 # we want to access low-level APIs.
575 loop = asyncio.get_running_loop()
576
577 # Create a pair of connected sockets.
578 rsock, wsock = socket.socketpair()
579
580 # Register the open socket to wait for data.
581 reader, writer = await asyncio.open_connection(sock=rsock)
582
583 # Simulate the reception of data from the network
584 loop.call_soon(wsock.send, 'abc'.encode())
585
586 # Wait for data
587 data = await reader.read(100)
588
589 # Got data, we are done: close the socket
590 print("Received:", data.decode())
591 writer.close()
592 await writer.wait_closed()
593
594 # Close the second socket
595 wsock.close()
596
597 asyncio.run(wait_for_data())
598
599 .. seealso::
600
601 The :ref:`register an open socket to wait for data using a protocol
602 <asyncio_example_create_connection>` example uses a low-level protocol and
603 the :meth:`loop.create_connection` method.
604
605 The :ref:`watch a file descriptor for read events
606 <asyncio_example_watch_fd>` example uses the low-level
607 :meth:`loop.add_reader` method to watch a file descriptor.