| 1 |
.. currentmodule:: asyncio |
| 2 |
|
| 3 |
.. _asyncio-queues: |
| 4 |
|
| 5 |
====== |
| 6 |
Queues |
| 7 |
====== |
| 8 |
|
| 9 |
**Source code:** :source:`Lib/asyncio/queues.py` |
| 10 |
|
| 11 |
------------------------------------------------ |
| 12 |
|
| 13 |
asyncio queues are designed to be similar to classes of the |
| 14 |
:mod:`queue` module. Although asyncio queues are not thread-safe, |
| 15 |
they are designed to be used specifically in async/await code. |
| 16 |
|
| 17 |
Note that methods of asyncio queues don't have a *timeout* parameter; |
| 18 |
use :func:`asyncio.wait_for` function to do queue operations with a |
| 19 |
timeout. |
| 20 |
|
| 21 |
See also the `Examples`_ section below. |
| 22 |
|
| 23 |
Queue |
| 24 |
===== |
| 25 |
|
| 26 |
.. class:: Queue(maxsize=0) |
| 27 |
|
| 28 |
A first in, first out (FIFO) queue. |
| 29 |
|
| 30 |
If *maxsize* is less than or equal to zero, the queue size is |
| 31 |
infinite. If it is an integer greater than ``0``, then |
| 32 |
``await put()`` blocks when the queue reaches *maxsize* |
| 33 |
until an item is removed by :meth:`get`. |
| 34 |
|
| 35 |
Unlike the standard library threading :mod:`queue`, the size of |
| 36 |
the queue is always known and can be returned by calling the |
| 37 |
:meth:`qsize` method. |
| 38 |
|
| 39 |
.. versionchanged:: 3.10 |
| 40 |
Removed the *loop* parameter. |
| 41 |
|
| 42 |
|
| 43 |
This class is :ref:`not thread safe <asyncio-multithreading>`. |
| 44 |
|
| 45 |
.. attribute:: maxsize |
| 46 |
|
| 47 |
Number of items allowed in the queue. |
| 48 |
|
| 49 |
.. method:: empty() |
| 50 |
|
| 51 |
Return ``True`` if the queue is empty, ``False`` otherwise. |
| 52 |
|
| 53 |
.. method:: full() |
| 54 |
|
| 55 |
Return ``True`` if there are :attr:`maxsize` items in the queue. |
| 56 |
|
| 57 |
If the queue was initialized with ``maxsize=0`` (the default), |
| 58 |
then :meth:`full` never returns ``True``. |
| 59 |
|
| 60 |
.. method:: get() |
| 61 |
:async: |
| 62 |
|
| 63 |
Remove and return an item from the queue. If queue is empty, |
| 64 |
wait until an item is available. |
| 65 |
|
| 66 |
Raises :exc:`QueueShutDown` if the queue has been shut down and |
| 67 |
is empty, or if the queue has been shut down immediately. |
| 68 |
|
| 69 |
.. method:: get_nowait() |
| 70 |
|
| 71 |
Return an item if one is immediately available, else raise |
| 72 |
:exc:`QueueEmpty`. |
| 73 |
|
| 74 |
.. method:: join() |
| 75 |
:async: |
| 76 |
|
| 77 |
Block until all items in the queue have been received and processed. |
| 78 |
|
| 79 |
The count of unfinished tasks goes up whenever an item is added |
| 80 |
to the queue. The count goes down whenever a consumer coroutine calls |
| 81 |
:meth:`task_done` to indicate that the item was retrieved and all |
| 82 |
work on it is complete. When the count of unfinished tasks drops |
| 83 |
to zero, :meth:`join` unblocks. |
| 84 |
|
| 85 |
.. method:: put(item) |
| 86 |
:async: |
| 87 |
|
| 88 |
Put an item into the queue. If the queue is full, wait until a |
| 89 |
free slot is available before adding the item. |
| 90 |
|
| 91 |
Raises :exc:`QueueShutDown` if the queue has been shut down. |
| 92 |
|
| 93 |
.. method:: put_nowait(item) |
| 94 |
|
| 95 |
Put an item into the queue without blocking. |
| 96 |
|
| 97 |
If no free slot is immediately available, raise :exc:`QueueFull`. |
| 98 |
|
| 99 |
.. method:: qsize() |
| 100 |
|
| 101 |
Return the number of items in the queue. |
| 102 |
|
| 103 |
.. method:: shutdown(immediate=False) |
| 104 |
|
| 105 |
Put a :class:`Queue` instance into a shutdown mode. |
| 106 |
|
| 107 |
The queue can no longer grow. |
| 108 |
Future calls to :meth:`~Queue.put` raise :exc:`QueueShutDown`. |
| 109 |
Currently blocked callers of :meth:`~Queue.put` will be unblocked |
| 110 |
and will raise :exc:`QueueShutDown` in the formerly awaiting task. |
| 111 |
|
| 112 |
If *immediate* is false (the default), the queue can be wound |
| 113 |
down normally with :meth:`~Queue.get` calls to extract tasks |
| 114 |
that have already been loaded. |
| 115 |
|
| 116 |
And if :meth:`~Queue.task_done` is called for each remaining task, a |
| 117 |
pending :meth:`~Queue.join` will be unblocked normally. |
| 118 |
|
| 119 |
Once the queue is empty, future calls to :meth:`~Queue.get` will |
| 120 |
raise :exc:`QueueShutDown`. |
| 121 |
|
| 122 |
If *immediate* is true, the queue is terminated immediately. |
| 123 |
The queue is drained to be completely empty and the count |
| 124 |
of unfinished tasks is reduced by the number of tasks drained. |
| 125 |
If unfinished tasks is zero, callers of :meth:`~Queue.join` |
| 126 |
are unblocked. Also, blocked callers of :meth:`~Queue.get` |
| 127 |
are unblocked and will raise :exc:`QueueShutDown` because the |
| 128 |
queue is empty. |
| 129 |
|
| 130 |
Use caution when using :meth:`~Queue.join` with *immediate* set |
| 131 |
to true. This unblocks the join even when no work has been done |
| 132 |
on the tasks, violating the usual invariant for joining a queue. |
| 133 |
|
| 134 |
.. versionadded:: 3.13 |
| 135 |
|
| 136 |
.. method:: task_done() |
| 137 |
|
| 138 |
Indicate that a formerly enqueued work item is complete. |
| 139 |
|
| 140 |
Used by queue consumers. For each :meth:`~Queue.get` used to |
| 141 |
fetch a work item, a subsequent call to :meth:`task_done` tells the |
| 142 |
queue that the processing on the work item is complete. |
| 143 |
|
| 144 |
If a :meth:`join` is currently blocking, it will resume when all |
| 145 |
items have been processed (meaning that a :meth:`task_done` |
| 146 |
call was received for every item that had been :meth:`~Queue.put` |
| 147 |
into the queue). |
| 148 |
|
| 149 |
Raises :exc:`ValueError` if called more times than there were |
| 150 |
items placed in the queue. |
| 151 |
|
| 152 |
|
| 153 |
Priority Queue |
| 154 |
============== |
| 155 |
|
| 156 |
.. class:: PriorityQueue |
| 157 |
|
| 158 |
A variant of :class:`Queue`; retrieves entries in priority order |
| 159 |
(lowest first). |
| 160 |
|
| 161 |
Entries are typically tuples of the form |
| 162 |
``(priority_number, data)``. |
| 163 |
|
| 164 |
|
| 165 |
LIFO Queue |
| 166 |
========== |
| 167 |
|
| 168 |
.. class:: LifoQueue |
| 169 |
|
| 170 |
A variant of :class:`Queue` that retrieves most recently added |
| 171 |
entries first (last in, first out). |
| 172 |
|
| 173 |
|
| 174 |
Exceptions |
| 175 |
========== |
| 176 |
|
| 177 |
.. exception:: QueueEmpty |
| 178 |
|
| 179 |
This exception is raised when the :meth:`~Queue.get_nowait` method |
| 180 |
is called on an empty queue. |
| 181 |
|
| 182 |
|
| 183 |
.. exception:: QueueFull |
| 184 |
|
| 185 |
Exception raised when the :meth:`~Queue.put_nowait` method is called |
| 186 |
on a queue that has reached its *maxsize*. |
| 187 |
|
| 188 |
|
| 189 |
.. exception:: QueueShutDown |
| 190 |
|
| 191 |
Exception raised when :meth:`~Queue.put` or :meth:`~Queue.get` is |
| 192 |
called on a queue which has been shut down. |
| 193 |
|
| 194 |
.. versionadded:: 3.13 |
| 195 |
|
| 196 |
|
| 197 |
Examples |
| 198 |
======== |
| 199 |
|
| 200 |
.. _asyncio_example_queue_dist: |
| 201 |
|
| 202 |
Queues can be used to distribute workload between several |
| 203 |
concurrent tasks:: |
| 204 |
|
| 205 |
import asyncio |
| 206 |
import random |
| 207 |
import time |
| 208 |
|
| 209 |
|
| 210 |
async def worker(name, queue): |
| 211 |
while True: |
| 212 |
# Get a "work item" out of the queue. |
| 213 |
sleep_for = await queue.get() |
| 214 |
|
| 215 |
# Sleep for the "sleep_for" seconds. |
| 216 |
await asyncio.sleep(sleep_for) |
| 217 |
|
| 218 |
# Notify the queue that the "work item" has been processed. |
| 219 |
queue.task_done() |
| 220 |
|
| 221 |
print(f'{name} has slept for {sleep_for:.2f} seconds') |
| 222 |
|
| 223 |
|
| 224 |
async def main(): |
| 225 |
# Create a queue that we will use to store our "workload". |
| 226 |
queue = asyncio.Queue() |
| 227 |
|
| 228 |
# Generate random timings and put them into the queue. |
| 229 |
total_sleep_time = 0 |
| 230 |
for _ in range(20): |
| 231 |
sleep_for = random.uniform(0.05, 1.0) |
| 232 |
total_sleep_time += sleep_for |
| 233 |
queue.put_nowait(sleep_for) |
| 234 |
|
| 235 |
# Create three worker tasks to process the queue concurrently. |
| 236 |
tasks = [] |
| 237 |
for i in range(3): |
| 238 |
task = asyncio.create_task(worker(f'worker-{i}', queue)) |
| 239 |
tasks.append(task) |
| 240 |
|
| 241 |
# Wait until the queue is fully processed. |
| 242 |
started_at = time.monotonic() |
| 243 |
await queue.join() |
| 244 |
total_slept_for = time.monotonic() - started_at |
| 245 |
|
| 246 |
# Cancel our worker tasks. |
| 247 |
for task in tasks: |
| 248 |
task.cancel() |
| 249 |
# Wait until all worker tasks are cancelled. |
| 250 |
await asyncio.gather(*tasks, return_exceptions=True) |
| 251 |
|
| 252 |
print('====') |
| 253 |
print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds') |
| 254 |
print(f'total expected sleep time: {total_sleep_time:.2f} seconds') |
| 255 |
|
| 256 |
|
| 257 |
asyncio.run(main()) |