Debian Codegraph @

Doc/library/queue.rst

raw
1 :mod:`!queue` --- A synchronized queue class
2 ============================================
3
4 .. module:: queue
5 :synopsis: A synchronized queue class.
6
7 **Source code:** :source:`Lib/queue.py`
8
9 --------------
10
11 The :mod:`!queue` module implements multi-producer, multi-consumer queues.
12 It is especially useful in threaded programming when information must be
13 exchanged safely between multiple threads. The :class:`Queue` class in this
14 module implements all the required locking semantics.
15
16 The module implements three types of queue, which differ only in the order in
17 which the entries are retrieved. In a :abbr:`FIFO (first-in, first-out)`
18 queue, the first tasks added are the first retrieved. In a
19 :abbr:`LIFO (last-in, first-out)` queue, the most recently added entry is
20 the first retrieved (operating like a stack). With a priority queue,
21 the entries are kept sorted (using the :mod:`heapq` module) and the
22 lowest valued entry is retrieved first.
23
24 Internally, those three types of queues use locks to temporarily block
25 competing threads; however, they are not designed to handle reentrancy
26 within a thread.
27
28 In addition, the module implements a "simple"
29 :abbr:`FIFO (first-in, first-out)` queue type, :class:`SimpleQueue`, whose
30 specific implementation provides additional guarantees
31 in exchange for the smaller functionality.
32
33 The :mod:`!queue` module defines the following classes and exceptions:
34
35 .. class:: Queue(maxsize=0)
36
37 Constructor for a :abbr:`FIFO (first-in, first-out)` queue. *maxsize* is
38 an integer that sets the upperbound
39 limit on the number of items that can be placed in the queue. Insertion will
40 block once this size has been reached, until queue items are consumed. If
41 *maxsize* is less than or equal to zero, the queue size is infinite.
42
43 .. class:: LifoQueue(maxsize=0)
44
45 Constructor for a :abbr:`LIFO (last-in, first-out)` queue. *maxsize* is
46 an integer that sets the upperbound
47 limit on the number of items that can be placed in the queue. Insertion will
48 block once this size has been reached, until queue items are consumed. If
49 *maxsize* is less than or equal to zero, the queue size is infinite.
50
51
52 .. class:: PriorityQueue(maxsize=0)
53
54 Constructor for a priority queue. *maxsize* is an integer that sets the upperbound
55 limit on the number of items that can be placed in the queue. Insertion will
56 block once this size has been reached, until queue items are consumed. If
57 *maxsize* is less than or equal to zero, the queue size is infinite.
58
59 The lowest valued entries are retrieved first (the lowest valued entry is the
60 one that would be returned by ``min(entries)``). A typical pattern for
61 entries is a tuple in the form: ``(priority_number, data)``.
62
63 If the *data* elements are not comparable, the data can be wrapped in a class
64 that ignores the data item and only compares the priority number::
65
66 from dataclasses import dataclass, field
67 from typing import Any
68
69 @dataclass(order=True)
70 class PrioritizedItem:
71 priority: int
72 item: Any=field(compare=False)
73
74 .. class:: SimpleQueue()
75
76 Constructor for an unbounded :abbr:`FIFO (first-in, first-out)` queue.
77 Simple queues lack advanced functionality such as task tracking.
78
79 .. versionadded:: 3.7
80
81
82 .. exception:: Empty
83
84 Exception raised when non-blocking :meth:`~Queue.get` (or
85 :meth:`~Queue.get_nowait`) is called
86 on a :class:`Queue` object which is empty.
87
88
89 .. exception:: Full
90
91 Exception raised when non-blocking :meth:`~Queue.put` (or
92 :meth:`~Queue.put_nowait`) is called
93 on a :class:`Queue` object which is full.
94
95
96 .. exception:: ShutDown
97
98 Exception raised when :meth:`~Queue.put` or :meth:`~Queue.get` is called on
99 a :class:`Queue` object which has been shut down.
100
101 .. versionadded:: 3.13
102
103
104 .. _queueobjects:
105
106 Queue Objects
107 -------------
108
109 Queue objects (:class:`Queue`, :class:`LifoQueue`, or :class:`PriorityQueue`)
110 provide the public methods described below.
111
112
113 .. method:: Queue.qsize()
114
115 Return the approximate size of the queue. Note, qsize() > 0 doesn't
116 guarantee that a subsequent get() will not block, nor will qsize() < maxsize
117 guarantee that put() will not block.
118
119
120 .. method:: Queue.empty()
121
122 Return ``True`` if the queue is empty, ``False`` otherwise. If empty()
123 returns ``True`` it doesn't guarantee that a subsequent call to put()
124 will not block. Similarly, if empty() returns ``False`` it doesn't
125 guarantee that a subsequent call to get() will not block.
126
127
128 .. method:: Queue.full()
129
130 Return ``True`` if the queue is full, ``False`` otherwise. If full()
131 returns ``True`` it doesn't guarantee that a subsequent call to get()
132 will not block. Similarly, if full() returns ``False`` it doesn't
133 guarantee that a subsequent call to put() will not block.
134
135
136 .. method:: Queue.put(item, block=True, timeout=None)
137
138 Put *item* into the queue. If optional args *block* is true and *timeout* is
139 ``None`` (the default), block if necessary until a free slot is available. If
140 *timeout* is a positive number, it blocks at most *timeout* seconds and raises
141 the :exc:`Full` exception if no free slot was available within that time.
142 Otherwise (*block* is false), put an item on the queue if a free slot is
143 immediately available, else raise the :exc:`Full` exception (*timeout* is
144 ignored in that case).
145
146 Raises :exc:`ShutDown` if the queue has been shut down.
147
148
149 .. method:: Queue.put_nowait(item)
150
151 Equivalent to ``put(item, block=False)``.
152
153
154 .. method:: Queue.get(block=True, timeout=None)
155
156 Remove and return an item from the queue. If optional args *block* is true and
157 *timeout* is ``None`` (the default), block if necessary until an item is available.
158 If *timeout* is a positive number, it blocks at most *timeout* seconds and
159 raises the :exc:`Empty` exception if no item was available within that time.
160 Otherwise (*block* is false), return an item if one is immediately available,
161 else raise the :exc:`Empty` exception (*timeout* is ignored in that case).
162
163 Prior to 3.0 on POSIX systems, and for all versions on Windows, if
164 *block* is true and *timeout* is ``None``, this operation goes into
165 an uninterruptible wait on an underlying lock. This means that no exceptions
166 can occur, and in particular a SIGINT will not trigger a :exc:`KeyboardInterrupt`.
167
168 Raises :exc:`ShutDown` if the queue has been shut down and is empty, or if
169 the queue has been shut down immediately.
170
171
172 .. method:: Queue.get_nowait()
173
174 Equivalent to ``get(False)``.
175
176 Two methods are offered to support tracking whether enqueued tasks have been
177 fully processed by daemon consumer threads.
178
179
180 .. method:: Queue.task_done()
181
182 Indicate that a formerly enqueued task is complete. Used by queue consumer
183 threads. For each :meth:`get` used to fetch a task, a subsequent call to
184 :meth:`task_done` tells the queue that the processing on the task is complete.
185
186 If a :meth:`join` is currently blocking, it will resume when all items have been
187 processed (meaning that a :meth:`task_done` call was received for every item
188 that had been :meth:`put` into the queue).
189
190 Raises a :exc:`ValueError` if called more times than there were items placed in
191 the queue.
192
193
194 .. method:: Queue.join()
195
196 Blocks until all items in the queue have been gotten and processed.
197
198 The count of unfinished tasks goes up whenever an item is added to the queue.
199 The count goes down whenever a consumer thread calls :meth:`task_done` to
200 indicate that the item was retrieved and all work on it is complete. When the
201 count of unfinished tasks drops to zero, :meth:`join` unblocks.
202
203
204 Waiting for task completion
205 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
206
207 Example of how to wait for enqueued tasks to be completed::
208
209 import threading
210 import queue
211
212 q = queue.Queue()
213
214 def worker():
215 while True:
216 item = q.get()
217 print(f'Working on {item}')
218 print(f'Finished {item}')
219 q.task_done()
220
221 # Turn-on the worker thread.
222 threading.Thread(target=worker, daemon=True).start()
223
224 # Send thirty task requests to the worker.
225 for item in range(30):
226 q.put(item)
227
228 # Block until all tasks are done.
229 q.join()
230 print('All work completed')
231
232
233 Terminating queues
234 ^^^^^^^^^^^^^^^^^^
235
236 When no longer needed, :class:`Queue` objects can be wound down
237 until empty or terminated immediately with a hard shutdown.
238
239 .. method:: Queue.shutdown(immediate=False)
240
241 Put a :class:`Queue` instance into a shutdown mode.
242
243 The queue can no longer grow.
244 Future calls to :meth:`~Queue.put` raise :exc:`ShutDown`.
245 Currently blocked callers of :meth:`~Queue.put` will be unblocked
246 and will raise :exc:`ShutDown` in the formerly blocked thread.
247
248 If *immediate* is false (the default), the queue can be wound
249 down normally with :meth:`~Queue.get` calls to extract tasks
250 that have already been loaded.
251
252 And if :meth:`~Queue.task_done` is called for each remaining task, a
253 pending :meth:`~Queue.join` will be unblocked normally.
254
255 Once the queue is empty, future calls to :meth:`~Queue.get` will
256 raise :exc:`ShutDown`.
257
258 If *immediate* is true, the queue is terminated immediately.
259 The queue is drained to be completely empty and the count
260 of unfinished tasks is reduced by the number of tasks drained.
261 If unfinished tasks is zero, callers of :meth:`~Queue.join`
262 are unblocked. Also, blocked callers of :meth:`~Queue.get`
263 are unblocked and will raise :exc:`ShutDown` because the
264 queue is empty.
265
266 Use caution when using :meth:`~Queue.join` with *immediate* set
267 to true. This unblocks the join even when no work has been done
268 on the tasks, violating the usual invariant for joining a queue.
269
270 .. versionadded:: 3.13
271
272
273 SimpleQueue Objects
274 -------------------
275
276 :class:`SimpleQueue` objects provide the public methods described below.
277
278 .. method:: SimpleQueue.qsize()
279
280 Return the approximate size of the queue. Note, qsize() > 0 doesn't
281 guarantee that a subsequent get() will not block.
282
283
284 .. method:: SimpleQueue.empty()
285
286 Return ``True`` if the queue is empty, ``False`` otherwise. If empty()
287 returns ``False`` it doesn't guarantee that a subsequent call to get()
288 will not block.
289
290
291 .. method:: SimpleQueue.put(item, block=True, timeout=None)
292
293 Put *item* into the queue. The method never blocks and always succeeds
294 (except for potential low-level errors such as failure to allocate memory).
295 The optional args *block* and *timeout* are ignored and only provided
296 for compatibility with :meth:`Queue.put`.
297
298 .. impl-detail::
299 This method has a C implementation which is reentrant. That is, a
300 ``put()`` or ``get()`` call can be interrupted by another ``put()``
301 call in the same thread without deadlocking or corrupting internal
302 state inside the queue. This makes it appropriate for use in
303 destructors such as ``__del__`` methods or :mod:`weakref` callbacks.
304
305
306 .. method:: SimpleQueue.put_nowait(item)
307
308 Equivalent to ``put(item, block=False)``, provided for compatibility with
309 :meth:`Queue.put_nowait`.
310
311
312 .. method:: SimpleQueue.get(block=True, timeout=None)
313
314 Remove and return an item from the queue. If optional args *block* is true and
315 *timeout* is ``None`` (the default), block if necessary until an item is available.
316 If *timeout* is a positive number, it blocks at most *timeout* seconds and
317 raises the :exc:`Empty` exception if no item was available within that time.
318 Otherwise (*block* is false), return an item if one is immediately available,
319 else raise the :exc:`Empty` exception (*timeout* is ignored in that case).
320
321
322 .. method:: SimpleQueue.get_nowait()
323
324 Equivalent to ``get(False)``.
325
326
327 .. seealso::
328
329 Class :class:`multiprocessing.Queue`
330 A queue class for use in a multi-processing (rather than multi-threading)
331 context.
332
333 :class:`collections.deque` is an alternative implementation of unbounded
334 queues with fast atomic :meth:`~collections.deque.append` and
335 :meth:`~collections.deque.popleft` operations that do not require locking
336 and also support indexing.