Debian Codegraph @

Doc/library/queue.rst

raw
1 :mod:`!queue` --- A synchronized queue class
2 ============================================
3
4 .. module:: queue
5 :synopsis: A synchronized queue class.
6
7 **Source code:** :source:`Lib/queue.py`
8
9 --------------
10
11 The :mod:`!queue` module implements multi-producer, multi-consumer queues.
12 It is especially useful in threaded programming when information must be
13 exchanged safely between multiple threads. The :class:`Queue` class in this
14 module implements all the required locking semantics.
15
16 The module implements three types of queue, which differ only in the order in
17 which the entries are retrieved. In a :abbr:`FIFO (first-in, first-out)`
18 queue, the first tasks added are the first retrieved. In a
19 :abbr:`LIFO (last-in, first-out)` queue, the most recently added entry is
20 the first retrieved (operating like a stack). With a priority queue,
21 the entries are kept sorted (using the :mod:`heapq` module) and the
22 lowest valued entry is retrieved first.
23
24 Internally, those three types of queues use locks to temporarily block
25 competing threads; however, they are not designed to handle reentrancy
26 within a thread.
27
28 In addition, the module implements a "simple"
29 :abbr:`FIFO (first-in, first-out)` queue type, :class:`SimpleQueue`, whose
30 specific implementation provides additional guarantees
31 in exchange for the smaller functionality.
32
33 The :mod:`!queue` module defines the following classes and exceptions:
34
35 .. class:: Queue(maxsize=0)
36
37 Constructor for a :abbr:`FIFO (first-in, first-out)` queue. *maxsize* is
38 an integer that sets the upperbound
39 limit on the number of items that can be placed in the queue. Insertion will
40 block once this size has been reached, until queue items are consumed. If
41 *maxsize* is less than or equal to zero, the queue size is infinite.
42
43 .. class:: LifoQueue(maxsize=0)
44
45 Constructor for a :abbr:`LIFO (last-in, first-out)` queue. *maxsize* is
46 an integer that sets the upperbound
47 limit on the number of items that can be placed in the queue. Insertion will
48 block once this size has been reached, until queue items are consumed. If
49 *maxsize* is less than or equal to zero, the queue size is infinite.
50
51
52 .. class:: PriorityQueue(maxsize=0)
53
54 Constructor for a priority queue. *maxsize* is an integer that sets the upperbound
55 limit on the number of items that can be placed in the queue. Insertion will
56 block once this size has been reached, until queue items are consumed. If
57 *maxsize* is less than or equal to zero, the queue size is infinite.
58
59 The lowest valued entries are retrieved first (the lowest valued entry is the
60 one that would be returned by ``min(entries)``). A typical pattern for
61 entries is a tuple in the form: ``(priority_number, data)``.
62
63 If the *data* elements are not comparable, the data can be wrapped in a class
64 that ignores the data item and only compares the priority number::
65
66 from dataclasses import dataclass, field
67 from typing import Any
68
69 @dataclass(order=True)
70 class PrioritizedItem:
71 priority: int
72 item: Any=field(compare=False)
73
74 .. class:: SimpleQueue()
75
76 Constructor for an unbounded :abbr:`FIFO (first-in, first-out)` queue.
77 Simple queues lack advanced functionality such as task tracking.
78
79 Simple queues are :ref:`generic <generics>` over the type of their items.
80
81 .. versionadded:: 3.7
82
83
84 .. exception:: Empty
85
86 Exception raised when non-blocking :meth:`~Queue.get` (or
87 :meth:`~Queue.get_nowait`) is called
88 on a :class:`Queue` object which is empty.
89
90
91 .. exception:: Full
92
93 Exception raised when non-blocking :meth:`~Queue.put` (or
94 :meth:`~Queue.put_nowait`) is called
95 on a :class:`Queue` object which is full.
96
97
98 .. exception:: ShutDown
99
100 Exception raised when :meth:`~Queue.put` or :meth:`~Queue.get` is called on
101 a :class:`Queue` object which has been shut down.
102
103 .. versionadded:: 3.13
104
105
106 .. _queueobjects:
107
108 Queue Objects
109 -------------
110
111 Queue objects (:class:`Queue`, :class:`LifoQueue`, or :class:`PriorityQueue`)
112 provide the public methods described below.
113
114
115 .. method:: Queue.qsize()
116
117 Return the approximate size of the queue. Note, qsize() > 0 doesn't
118 guarantee that a subsequent get() will not block, nor will qsize() < maxsize
119 guarantee that put() will not block.
120
121
122 .. method:: Queue.empty()
123
124 Return ``True`` if the queue is empty, ``False`` otherwise. If empty()
125 returns ``True`` it doesn't guarantee that a subsequent call to put()
126 will not block. Similarly, if empty() returns ``False`` it doesn't
127 guarantee that a subsequent call to get() will not block.
128
129
130 .. method:: Queue.full()
131
132 Return ``True`` if the queue is full, ``False`` otherwise. If full()
133 returns ``True`` it doesn't guarantee that a subsequent call to get()
134 will not block. Similarly, if full() returns ``False`` it doesn't
135 guarantee that a subsequent call to put() will not block.
136
137
138 .. method:: Queue.put(item, block=True, timeout=None)
139
140 Put *item* into the queue. If optional args *block* is true and *timeout* is
141 ``None`` (the default), block if necessary until a free slot is available. If
142 *timeout* is a positive number, it blocks at most *timeout* seconds and raises
143 the :exc:`Full` exception if no free slot was available within that time.
144 Otherwise (*block* is false), put an item on the queue if a free slot is
145 immediately available, else raise the :exc:`Full` exception (*timeout* is
146 ignored in that case).
147
148 Raises :exc:`ShutDown` if the queue has been shut down.
149
150
151 .. method:: Queue.put_nowait(item)
152
153 Equivalent to ``put(item, block=False)``.
154
155
156 .. method:: Queue.get(block=True, timeout=None)
157
158 Remove and return an item from the queue. If optional args *block* is true and
159 *timeout* is ``None`` (the default), block if necessary until an item is available.
160 If *timeout* is a positive number, it blocks at most *timeout* seconds and
161 raises the :exc:`Empty` exception if no item was available within that time.
162 Otherwise (*block* is false), return an item if one is immediately available,
163 else raise the :exc:`Empty` exception (*timeout* is ignored in that case).
164
165 Prior to 3.0 on POSIX systems, and for all versions on Windows, if
166 *block* is true and *timeout* is ``None``, this operation goes into
167 an uninterruptible wait on an underlying lock. This means that no exceptions
168 can occur, and in particular a SIGINT will not trigger a :exc:`KeyboardInterrupt`.
169
170 Raises :exc:`ShutDown` if the queue has been shut down and is empty, or if
171 the queue has been shut down immediately.
172
173
174 .. method:: Queue.get_nowait()
175
176 Equivalent to ``get(False)``.
177
178 Two methods are offered to support tracking whether enqueued tasks have been
179 fully processed by daemon consumer threads.
180
181
182 .. method:: Queue.task_done()
183
184 Indicate that a formerly enqueued task is complete. Used by queue consumer
185 threads. For each :meth:`get` used to fetch a task, a subsequent call to
186 :meth:`task_done` tells the queue that the processing on the task is complete.
187
188 If a :meth:`join` is currently blocking, it will resume when all items have been
189 processed (meaning that a :meth:`task_done` call was received for every item
190 that had been :meth:`put` into the queue).
191
192 Raises a :exc:`ValueError` if called more times than there were items placed in
193 the queue.
194
195
196 .. method:: Queue.join()
197
198 Blocks until all items in the queue have been gotten and processed.
199
200 The count of unfinished tasks goes up whenever an item is added to the queue.
201 The count goes down whenever a consumer thread calls :meth:`task_done` to
202 indicate that the item was retrieved and all work on it is complete. When the
203 count of unfinished tasks drops to zero, :meth:`join` unblocks.
204
205
206 Waiting for task completion
207 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
208
209 Example of how to wait for enqueued tasks to be completed::
210
211 import threading
212 import queue
213
214 q = queue.Queue()
215
216 def worker():
217 while True:
218 item = q.get()
219 print(f'Working on {item}')
220 print(f'Finished {item}')
221 q.task_done()
222
223 # Turn-on the worker thread.
224 threading.Thread(target=worker, daemon=True).start()
225
226 # Send thirty task requests to the worker.
227 for item in range(30):
228 q.put(item)
229
230 # Block until all tasks are done.
231 q.join()
232 print('All work completed')
233
234
235 Terminating queues
236 ^^^^^^^^^^^^^^^^^^
237
238 When no longer needed, :class:`Queue` objects can be wound down
239 until empty or terminated immediately with a hard shutdown.
240
241 .. method:: Queue.shutdown(immediate=False)
242
243 Put a :class:`Queue` instance into a shutdown mode.
244
245 The queue can no longer grow.
246 Future calls to :meth:`~Queue.put` raise :exc:`ShutDown`.
247 Currently blocked callers of :meth:`~Queue.put` will be unblocked
248 and will raise :exc:`ShutDown` in the formerly blocked thread.
249
250 If *immediate* is false (the default), the queue can be wound
251 down normally with :meth:`~Queue.get` calls to extract tasks
252 that have already been loaded.
253
254 And if :meth:`~Queue.task_done` is called for each remaining task, a
255 pending :meth:`~Queue.join` will be unblocked normally.
256
257 Once the queue is empty, future calls to :meth:`~Queue.get` will
258 raise :exc:`ShutDown`.
259
260 If *immediate* is true, the queue is terminated immediately.
261 The queue is drained to be completely empty and the count
262 of unfinished tasks is reduced by the number of tasks drained.
263 If unfinished tasks is zero, callers of :meth:`~Queue.join`
264 are unblocked. Also, blocked callers of :meth:`~Queue.get`
265 are unblocked and will raise :exc:`ShutDown` because the
266 queue is empty.
267
268 Use caution when using :meth:`~Queue.join` with *immediate* set
269 to true. This unblocks the join even when no work has been done
270 on the tasks, violating the usual invariant for joining a queue.
271
272 .. versionadded:: 3.13
273
274
275 SimpleQueue Objects
276 -------------------
277
278 :class:`SimpleQueue` objects provide the public methods described below.
279
280 .. method:: SimpleQueue.qsize()
281
282 Return the approximate size of the queue. Note, qsize() > 0 doesn't
283 guarantee that a subsequent get() will not block.
284
285
286 .. method:: SimpleQueue.empty()
287
288 Return ``True`` if the queue is empty, ``False`` otherwise. If empty()
289 returns ``False`` it doesn't guarantee that a subsequent call to get()
290 will not block.
291
292
293 .. method:: SimpleQueue.put(item, block=True, timeout=None)
294
295 Put *item* into the queue. The method never blocks and always succeeds
296 (except for potential low-level errors such as failure to allocate memory).
297 The optional args *block* and *timeout* are ignored and only provided
298 for compatibility with :meth:`Queue.put`.
299
300 .. impl-detail::
301 This method has a C implementation which is reentrant. That is, a
302 ``put()`` or ``get()`` call can be interrupted by another ``put()``
303 call in the same thread without deadlocking or corrupting internal
304 state inside the queue. This makes it appropriate for use in
305 destructors such as ``__del__`` methods or :mod:`weakref` callbacks.
306
307
308 .. method:: SimpleQueue.put_nowait(item)
309
310 Equivalent to ``put(item, block=False)``, provided for compatibility with
311 :meth:`Queue.put_nowait`.
312
313
314 .. method:: SimpleQueue.get(block=True, timeout=None)
315
316 Remove and return an item from the queue. If optional args *block* is true and
317 *timeout* is ``None`` (the default), block if necessary until an item is available.
318 If *timeout* is a positive number, it blocks at most *timeout* seconds and
319 raises the :exc:`Empty` exception if no item was available within that time.
320 Otherwise (*block* is false), return an item if one is immediately available,
321 else raise the :exc:`Empty` exception (*timeout* is ignored in that case).
322
323
324 .. method:: SimpleQueue.get_nowait()
325
326 Equivalent to ``get(False)``.
327
328
329 .. seealso::
330
331 Class :class:`multiprocessing.Queue`
332 A queue class for use in a multi-processing (rather than multi-threading)
333 context.
334
335 :class:`collections.deque` is an alternative implementation of unbounded
336 queues with fast atomic :meth:`~collections.deque.append` and
337 :meth:`~collections.deque.popleft` operations that do not require locking
338 and also support indexing.