Debian Codegraph @

Doc/library/asyncio-queue.rst

raw
1 .. currentmodule:: asyncio
2
3 .. _asyncio-queues:
4
5 ======
6 Queues
7 ======
8
9 **Source code:** :source:`Lib/asyncio/queues.py`
10
11 ------------------------------------------------
12
13 asyncio queues are designed to be similar to classes of the
14 :mod:`queue` module. Although asyncio queues are not thread-safe,
15 they are designed to be used specifically in async/await code.
16
17 Note that methods of asyncio queues don't have a *timeout* parameter;
18 use :func:`asyncio.wait_for` function to do queue operations with a
19 timeout.
20
21 See also the `Examples`_ section below.
22
23 Queue
24 =====
25
26 .. class:: Queue(maxsize=0)
27
28 A first in, first out (FIFO) queue.
29
30 If *maxsize* is less than or equal to zero, the queue size is
31 infinite. If it is an integer greater than ``0``, then
32 ``await put()`` blocks when the queue reaches *maxsize*
33 until an item is removed by :meth:`get`.
34
35 Unlike the standard library threading :mod:`queue`, the size of
36 the queue is always known and can be returned by calling the
37 :meth:`qsize` method.
38
39 .. versionchanged:: 3.10
40 Removed the *loop* parameter.
41
42
43 This class is :ref:`not thread safe <asyncio-multithreading>`.
44
45 .. attribute:: maxsize
46
47 Number of items allowed in the queue.
48
49 .. method:: empty()
50
51 Return ``True`` if the queue is empty, ``False`` otherwise.
52
53 .. method:: full()
54
55 Return ``True`` if there are :attr:`maxsize` items in the queue.
56
57 If the queue was initialized with ``maxsize=0`` (the default),
58 then :meth:`full` never returns ``True``.
59
60 .. method:: get()
61 :async:
62
63 Remove and return an item from the queue. If queue is empty,
64 wait until an item is available.
65
66 Raises :exc:`QueueShutDown` if the queue has been shut down and
67 is empty, or if the queue has been shut down immediately.
68
69 .. method:: get_nowait()
70
71 Return an item if one is immediately available, else raise
72 :exc:`QueueEmpty`.
73
74 Raises :exc:`QueueShutDown` if the queue has been shut down and is empty.
75
76 .. method:: join()
77 :async:
78
79 Block until all items in the queue have been received and processed.
80
81 The count of unfinished tasks goes up whenever an item is added
82 to the queue. The count goes down whenever a consumer coroutine calls
83 :meth:`task_done` to indicate that the item was retrieved and all
84 work on it is complete. When the count of unfinished tasks drops
85 to zero, :meth:`join` unblocks.
86
87 .. method:: put(item)
88 :async:
89
90 Put an item into the queue. If the queue is full, wait until a
91 free slot is available before adding the item.
92
93 Raises :exc:`QueueShutDown` if the queue has been shut down.
94
95 .. method:: put_nowait(item)
96
97 Put an item into the queue without blocking.
98
99 If no free slot is immediately available, raise :exc:`QueueFull`.
100
101 Raises :exc:`QueueShutDown` if the queue has been shut down.
102
103 .. method:: qsize()
104
105 Return the number of items in the queue.
106
107 .. method:: shutdown(immediate=False)
108
109 Put a :class:`Queue` instance into a shutdown mode.
110
111 The queue can no longer grow.
112 Future calls to :meth:`~Queue.put` raise :exc:`QueueShutDown`.
113 Currently blocked callers of :meth:`~Queue.put` will be unblocked
114 and will raise :exc:`QueueShutDown` in the formerly awaiting task.
115
116 If *immediate* is false (the default), the queue can be wound
117 down normally with :meth:`~Queue.get` calls to extract tasks
118 that have already been loaded.
119
120 And if :meth:`~Queue.task_done` is called for each remaining task, a
121 pending :meth:`~Queue.join` will be unblocked normally.
122
123 Once the queue is empty, future calls to :meth:`~Queue.get` will
124 raise :exc:`QueueShutDown`.
125
126 If *immediate* is true, the queue is terminated immediately.
127 The queue is drained to be completely empty and the count
128 of unfinished tasks is reduced by the number of tasks drained.
129 If unfinished tasks is zero, callers of :meth:`~Queue.join`
130 are unblocked. Also, blocked callers of :meth:`~Queue.get`
131 are unblocked and will raise :exc:`QueueShutDown` because the
132 queue is empty.
133
134 Use caution when using :meth:`~Queue.join` with *immediate* set
135 to true. This unblocks the join even when no work has been done
136 on the tasks, violating the usual invariant for joining a queue.
137
138 .. versionadded:: 3.13
139
140 .. method:: task_done()
141
142 Indicate that a formerly enqueued work item is complete.
143
144 Used by queue consumers. For each :meth:`~Queue.get` used to
145 fetch a work item, a subsequent call to :meth:`task_done` tells the
146 queue that the processing on the work item is complete.
147
148 If a :meth:`join` is currently blocking, it will resume when all
149 items have been processed (meaning that a :meth:`task_done`
150 call was received for every item that had been :meth:`~Queue.put`
151 into the queue).
152
153 Raises :exc:`ValueError` if called more times than there were
154 items placed in the queue.
155
156
157 Priority Queue
158 ==============
159
160 .. class:: PriorityQueue
161
162 A variant of :class:`Queue`; retrieves entries in priority order
163 (lowest first).
164
165 Entries are typically tuples of the form
166 ``(priority_number, data)``.
167
168
169 LIFO Queue
170 ==========
171
172 .. class:: LifoQueue
173
174 A variant of :class:`Queue` that retrieves most recently added
175 entries first (last in, first out).
176
177
178 Exceptions
179 ==========
180
181 .. exception:: QueueEmpty
182
183 This exception is raised when the :meth:`~Queue.get_nowait` method
184 is called on an empty queue.
185
186
187 .. exception:: QueueFull
188
189 Exception raised when the :meth:`~Queue.put_nowait` method is called
190 on a queue that has reached its *maxsize*.
191
192
193 .. exception:: QueueShutDown
194
195 Exception raised when :meth:`~Queue.put`, :meth:`~Queue.put_nowait`,
196 :meth:`~Queue.get` or :meth:`~Queue.get_nowait` is called
197 on a queue which has been shut down.
198
199 .. versionadded:: 3.13
200
201
202 Examples
203 ========
204
205 .. _asyncio_example_queue_dist:
206
207 Queues can be used to distribute workload between several
208 concurrent tasks::
209
210 import asyncio
211 import random
212 import time
213
214
215 async def worker(name, queue):
216 while True:
217 # Get a "work item" out of the queue.
218 sleep_for = await queue.get()
219
220 # Sleep for the "sleep_for" seconds.
221 await asyncio.sleep(sleep_for)
222
223 # Notify the queue that the "work item" has been processed.
224 queue.task_done()
225
226 print(f'{name} has slept for {sleep_for:.2f} seconds')
227
228
229 async def main():
230 # Create a queue that we will use to store our "workload".
231 queue = asyncio.Queue()
232
233 # Generate random timings and put them into the queue.
234 total_sleep_time = 0
235 for _ in range(20):
236 sleep_for = random.uniform(0.05, 1.0)
237 total_sleep_time += sleep_for
238 queue.put_nowait(sleep_for)
239
240 # Create three worker tasks to process the queue concurrently.
241 tasks = []
242 for i in range(3):
243 task = asyncio.create_task(worker(f'worker-{i}', queue))
244 tasks.append(task)
245
246 # Wait until the queue is fully processed.
247 started_at = time.monotonic()
248 await queue.join()
249 total_slept_for = time.monotonic() - started_at
250
251 # Cancel our worker tasks.
252 for task in tasks:
253 task.cancel()
254 # Wait until all worker tasks are cancelled.
255 await asyncio.gather(*tasks, return_exceptions=True)
256
257 print('====')
258 print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds')
259 print(f'total expected sleep time: {total_sleep_time:.2f} seconds')
260
261
262 asyncio.run(main())