Debian Codegraph @

Doc/library/asyncio-queue.rst

raw
1 .. currentmodule:: asyncio
2
3 .. _asyncio-queues:
4
5 ======
6 Queues
7 ======
8
9 **Source code:** :source:`Lib/asyncio/queues.py`
10
11 ------------------------------------------------
12
13 asyncio queues are designed to be similar to classes of the
14 :mod:`queue` module. Although asyncio queues are not thread-safe,
15 they are designed to be used specifically in async/await code.
16
17 Note that methods of asyncio queues don't have a *timeout* parameter;
18 use :func:`asyncio.wait_for` function to do queue operations with a
19 timeout.
20
21 See also the `Examples`_ section below.
22
23 Queue
24 =====
25
26 .. class:: Queue(maxsize=0)
27
28 A first in, first out (FIFO) queue.
29
30 If *maxsize* is less than or equal to zero, the queue size is
31 infinite. If it is an integer greater than ``0``, then
32 ``await put()`` blocks when the queue reaches *maxsize*
33 until an item is removed by :meth:`get`.
34
35 Unlike the standard library threading :mod:`queue`, the size of
36 the queue is always known and can be returned by calling the
37 :meth:`qsize` method.
38
39 .. versionchanged:: 3.10
40 Removed the *loop* parameter.
41
42
43 This class is :ref:`not thread safe <asyncio-multithreading>`.
44
45 .. attribute:: maxsize
46
47 Number of items allowed in the queue.
48
49 .. method:: empty()
50
51 Return ``True`` if the queue is empty, ``False`` otherwise.
52
53 .. method:: full()
54
55 Return ``True`` if there are :attr:`maxsize` items in the queue.
56
57 If the queue was initialized with ``maxsize=0`` (the default),
58 then :meth:`full` never returns ``True``.
59
60 .. method:: get()
61 :async:
62
63 Remove and return an item from the queue. If queue is empty,
64 wait until an item is available.
65
66 Raises :exc:`QueueShutDown` if the queue has been shut down and
67 is empty, or if the queue has been shut down immediately.
68
69 .. method:: get_nowait()
70
71 Return an item if one is immediately available, else raise
72 :exc:`QueueEmpty`.
73
74 .. method:: join()
75 :async:
76
77 Block until all items in the queue have been received and processed.
78
79 The count of unfinished tasks goes up whenever an item is added
80 to the queue. The count goes down whenever a consumer coroutine calls
81 :meth:`task_done` to indicate that the item was retrieved and all
82 work on it is complete. When the count of unfinished tasks drops
83 to zero, :meth:`join` unblocks.
84
85 .. method:: put(item)
86 :async:
87
88 Put an item into the queue. If the queue is full, wait until a
89 free slot is available before adding the item.
90
91 Raises :exc:`QueueShutDown` if the queue has been shut down.
92
93 .. method:: put_nowait(item)
94
95 Put an item into the queue without blocking.
96
97 If no free slot is immediately available, raise :exc:`QueueFull`.
98
99 .. method:: qsize()
100
101 Return the number of items in the queue.
102
103 .. method:: shutdown(immediate=False)
104
105 Put a :class:`Queue` instance into a shutdown mode.
106
107 The queue can no longer grow.
108 Future calls to :meth:`~Queue.put` raise :exc:`QueueShutDown`.
109 Currently blocked callers of :meth:`~Queue.put` will be unblocked
110 and will raise :exc:`QueueShutDown` in the formerly awaiting task.
111
112 If *immediate* is false (the default), the queue can be wound
113 down normally with :meth:`~Queue.get` calls to extract tasks
114 that have already been loaded.
115
116 And if :meth:`~Queue.task_done` is called for each remaining task, a
117 pending :meth:`~Queue.join` will be unblocked normally.
118
119 Once the queue is empty, future calls to :meth:`~Queue.get` will
120 raise :exc:`QueueShutDown`.
121
122 If *immediate* is true, the queue is terminated immediately.
123 The queue is drained to be completely empty and the count
124 of unfinished tasks is reduced by the number of tasks drained.
125 If unfinished tasks is zero, callers of :meth:`~Queue.join`
126 are unblocked. Also, blocked callers of :meth:`~Queue.get`
127 are unblocked and will raise :exc:`QueueShutDown` because the
128 queue is empty.
129
130 Use caution when using :meth:`~Queue.join` with *immediate* set
131 to true. This unblocks the join even when no work has been done
132 on the tasks, violating the usual invariant for joining a queue.
133
134 .. versionadded:: 3.13
135
136 .. method:: task_done()
137
138 Indicate that a formerly enqueued work item is complete.
139
140 Used by queue consumers. For each :meth:`~Queue.get` used to
141 fetch a work item, a subsequent call to :meth:`task_done` tells the
142 queue that the processing on the work item is complete.
143
144 If a :meth:`join` is currently blocking, it will resume when all
145 items have been processed (meaning that a :meth:`task_done`
146 call was received for every item that had been :meth:`~Queue.put`
147 into the queue).
148
149 Raises :exc:`ValueError` if called more times than there were
150 items placed in the queue.
151
152
153 Priority Queue
154 ==============
155
156 .. class:: PriorityQueue
157
158 A variant of :class:`Queue`; retrieves entries in priority order
159 (lowest first).
160
161 Entries are typically tuples of the form
162 ``(priority_number, data)``.
163
164
165 LIFO Queue
166 ==========
167
168 .. class:: LifoQueue
169
170 A variant of :class:`Queue` that retrieves most recently added
171 entries first (last in, first out).
172
173
174 Exceptions
175 ==========
176
177 .. exception:: QueueEmpty
178
179 This exception is raised when the :meth:`~Queue.get_nowait` method
180 is called on an empty queue.
181
182
183 .. exception:: QueueFull
184
185 Exception raised when the :meth:`~Queue.put_nowait` method is called
186 on a queue that has reached its *maxsize*.
187
188
189 .. exception:: QueueShutDown
190
191 Exception raised when :meth:`~Queue.put` or :meth:`~Queue.get` is
192 called on a queue which has been shut down.
193
194 .. versionadded:: 3.13
195
196
197 Examples
198 ========
199
200 .. _asyncio_example_queue_dist:
201
202 Queues can be used to distribute workload between several
203 concurrent tasks::
204
205 import asyncio
206 import random
207 import time
208
209
210 async def worker(name, queue):
211 while True:
212 # Get a "work item" out of the queue.
213 sleep_for = await queue.get()
214
215 # Sleep for the "sleep_for" seconds.
216 await asyncio.sleep(sleep_for)
217
218 # Notify the queue that the "work item" has been processed.
219 queue.task_done()
220
221 print(f'{name} has slept for {sleep_for:.2f} seconds')
222
223
224 async def main():
225 # Create a queue that we will use to store our "workload".
226 queue = asyncio.Queue()
227
228 # Generate random timings and put them into the queue.
229 total_sleep_time = 0
230 for _ in range(20):
231 sleep_for = random.uniform(0.05, 1.0)
232 total_sleep_time += sleep_for
233 queue.put_nowait(sleep_for)
234
235 # Create three worker tasks to process the queue concurrently.
236 tasks = []
237 for i in range(3):
238 task = asyncio.create_task(worker(f'worker-{i}', queue))
239 tasks.append(task)
240
241 # Wait until the queue is fully processed.
242 started_at = time.monotonic()
243 await queue.join()
244 total_slept_for = time.monotonic() - started_at
245
246 # Cancel our worker tasks.
247 for task in tasks:
248 task.cancel()
249 # Wait until all worker tasks are cancelled.
250 await asyncio.gather(*tasks, return_exceptions=True)
251
252 print('====')
253 print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds')
254 print(f'total expected sleep time: {total_sleep_time:.2f} seconds')
255
256
257 asyncio.run(main())