Debian Codegraph @

Doc/library/multiprocessing.shared_memory.rst

raw
1 :mod:`!multiprocessing.shared_memory` --- Shared memory for direct access across processes
2 ==========================================================================================
3
4 .. module:: multiprocessing.shared_memory
5 :synopsis: Provides shared memory for direct access across processes.
6
7 **Source code:** :source:`Lib/multiprocessing/shared_memory.py`
8
9 .. versionadded:: 3.8
10
11 .. index::
12 single: Shared Memory
13 single: POSIX Shared Memory
14 single: Named Shared Memory
15
16 --------------
17
18 This module provides a class, :class:`SharedMemory`, for the allocation
19 and management of shared memory to be accessed by one or more processes
20 on a multicore or symmetric multiprocessor (SMP) machine. To assist with
21 the life-cycle management of shared memory especially across distinct
22 processes, a :class:`~multiprocessing.managers.BaseManager` subclass,
23 :class:`~multiprocessing.managers.SharedMemoryManager`, is also provided in the
24 :mod:`multiprocessing.managers` module.
25
26 In this module, shared memory refers to "POSIX style" shared memory blocks
27 (though is not necessarily implemented explicitly as such) and does not refer
28 to "distributed shared memory". This style of shared memory permits distinct
29 processes to potentially read and write to a common (or shared) region of
30 volatile memory. Processes are conventionally limited to only have access to
31 their own process memory space but shared memory permits the sharing
32 of data between processes, avoiding the need to instead send messages between
33 processes containing that data. Sharing data directly via memory can provide
34 significant performance benefits compared to sharing data via disk or socket
35 or other communications requiring the serialization/deserialization and
36 copying of data.
37
38
39 .. class:: SharedMemory(name=None, create=False, size=0, *, track=True)
40
41 Create an instance of the :class:`!SharedMemory` class for either
42 creating a new shared memory block or attaching to an existing shared
43 memory block. Each shared memory block is assigned a unique name.
44 In this way, one process can create a shared memory block with a
45 particular name and a different process can attach to that same shared
46 memory block using that same name.
47
48 As a resource for sharing data across processes, shared memory blocks
49 may outlive the original process that created them. When one process
50 no longer needs access to a shared memory block that might still be
51 needed by other processes, the :meth:`close` method should be called.
52 When a shared memory block is no longer needed by any process, the
53 :meth:`unlink` method should be called to ensure proper cleanup.
54
55 :param name:
56 The unique name for the requested shared memory, specified as a string.
57 When creating a new shared memory block, if ``None`` (the default)
58 is supplied for the name, a novel name will be generated.
59 :type name: str | None
60
61 :param bool create:
62 Control whether a new shared memory block is created (``True``)
63 or an existing shared memory block is attached (``False``).
64
65 :param int size:
66 The requested number of bytes when creating a new shared memory block.
67 Because some platforms choose to allocate chunks of memory
68 based upon that platform's memory page size, the exact size of the shared
69 memory block may be larger or equal to the size requested.
70 When attaching to an existing shared memory block,
71 the *size* parameter is ignored.
72
73 :param bool track:
74 When ``True``, register the shared memory block with a resource
75 tracker process on platforms where the OS does not do this automatically.
76 The resource tracker ensures proper cleanup of the shared memory even
77 if all other processes with access to the memory exit without doing so.
78 Python processes created from a common ancestor using :mod:`multiprocessing`
79 facilities share a single resource tracker process, and the lifetime of
80 shared memory segments is handled automatically among these processes.
81 Python processes created in any other way will receive their own
82 resource tracker when accessing shared memory with *track* enabled.
83 This will cause the shared memory to be deleted by the resource tracker
84 of the first process that terminates.
85 To avoid this issue, users of :mod:`subprocess` or standalone Python
86 processes should set *track* to ``False`` when there is already another
87 process in place that does the bookkeeping.
88 *track* is ignored on Windows, which has its own tracking and
89 automatically deletes shared memory when all handles to it have been closed.
90
91 .. versionchanged:: 3.13
92 Added the *track* parameter.
93
94 .. method:: close()
95
96 Close the file descriptor/handle to the shared memory from this
97 instance. :meth:`close` should be called once access to the shared
98 memory block from this instance is no longer needed. Depending
99 on operating system, the underlying memory may or may not be freed
100 even if all handles to it have been closed. To ensure proper cleanup,
101 use the :meth:`unlink` method.
102
103 .. method:: unlink()
104
105 Delete the underlying shared memory block. This should be called only
106 once per shared memory block regardless of the number of handles to it,
107 even in other processes.
108 :meth:`unlink` and :meth:`close` can be called in any order, but
109 trying to access data inside a shared memory block after :meth:`unlink`
110 may result in memory access errors, depending on platform.
111
112 This method has no effect on Windows, where the only way to delete a
113 shared memory block is to close all handles.
114
115 .. attribute:: buf
116
117 A memoryview of contents of the shared memory block.
118
119 .. attribute:: name
120
121 Read-only access to the unique name of the shared memory block.
122
123 .. attribute:: size
124
125 Read-only access to size in bytes of the shared memory block.
126
127
128 The following example demonstrates low-level use of :class:`SharedMemory`
129 instances::
130
131 >>> from multiprocessing import shared_memory
132 >>> shm_a = shared_memory.SharedMemory(create=True, size=10)
133 >>> type(shm_a.buf)
134 <class 'memoryview'>
135 >>> buffer = shm_a.buf
136 >>> len(buffer)
137 10
138 >>> buffer[:4] = bytearray([22, 33, 44, 55]) # Modify multiple at once
139 >>> buffer[4] = 100 # Modify single byte at a time
140 >>> # Attach to an existing shared memory block
141 >>> shm_b = shared_memory.SharedMemory(shm_a.name)
142 >>> import array
143 >>> array.array('b', shm_b.buf[:5]) # Copy the data into a new array.array
144 array('b', [22, 33, 44, 55, 100])
145 >>> shm_b.buf[:5] = b'howdy' # Modify via shm_b using bytes
146 >>> bytes(shm_a.buf[:5]) # Access via shm_a
147 b'howdy'
148 >>> shm_b.close() # Close each SharedMemory instance
149 >>> shm_a.close()
150 >>> shm_a.unlink() # Call unlink only once to release the shared memory
151
152
153
154 The following example demonstrates a practical use of the :class:`SharedMemory`
155 class with `NumPy arrays <https://numpy.org/>`_, accessing the
156 same :class:`!numpy.ndarray` from two distinct Python shells:
157
158 .. doctest::
159 :options: +SKIP
160
161 >>> # In the first Python interactive shell
162 >>> import numpy as np
163 >>> a = np.array([1, 1, 2, 3, 5, 8]) # Start with an existing NumPy array
164 >>> from multiprocessing import shared_memory
165 >>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
166 >>> # Now create a NumPy array backed by shared memory
167 >>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
168 >>> b[:] = a[:] # Copy the original data into shared memory
169 >>> b
170 array([1, 1, 2, 3, 5, 8])
171 >>> type(b)
172 <class 'numpy.ndarray'>
173 >>> type(a)
174 <class 'numpy.ndarray'>
175 >>> shm.name # We did not specify a name so one was chosen for us
176 'psm_21467_46075'
177
178 >>> # In either the same shell or a new Python shell on the same machine
179 >>> import numpy as np
180 >>> from multiprocessing import shared_memory
181 >>> # Attach to the existing shared memory block
182 >>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075')
183 >>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example
184 >>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf)
185 >>> c
186 array([1, 1, 2, 3, 5, 8])
187 >>> c[-1] = 888
188 >>> c
189 array([ 1, 1, 2, 3, 5, 888])
190
191 >>> # Back in the first Python interactive shell, b reflects this change
192 >>> b
193 array([ 1, 1, 2, 3, 5, 888])
194
195 >>> # Clean up from within the second Python shell
196 >>> del c # Unnecessary; merely emphasizing the array is no longer used
197 >>> existing_shm.close()
198
199 >>> # Clean up from within the first Python shell
200 >>> del b # Unnecessary; merely emphasizing the array is no longer used
201 >>> shm.close()
202 >>> shm.unlink() # Free and release the shared memory block at the very end
203
204
205 .. class:: SharedMemoryManager([address[, authkey]])
206 :module: multiprocessing.managers
207
208 A subclass of :class:`multiprocessing.managers.BaseManager` which can be
209 used for the management of shared memory blocks across processes.
210
211 A call to :meth:`~multiprocessing.managers.BaseManager.start` on a
212 :class:`!SharedMemoryManager` instance causes a new process to be started.
213 This new process's sole purpose is to manage the life cycle
214 of all shared memory blocks created through it. To trigger the release
215 of all shared memory blocks managed by that process, call
216 :meth:`~multiprocessing.managers.BaseManager.shutdown` on the instance.
217 This triggers a :meth:`~multiprocessing.shared_memory.SharedMemory.unlink` call
218 on all of the :class:`SharedMemory` objects managed by that process and then
219 stops the process itself. By creating :class:`!SharedMemory` instances
220 through a :class:`!SharedMemoryManager`, we avoid the need to manually track
221 and trigger the freeing of shared memory resources.
222
223 This class provides methods for creating and returning :class:`SharedMemory`
224 instances and for creating a list-like object (:class:`ShareableList`)
225 backed by shared memory.
226
227 Refer to :class:`~multiprocessing.managers.BaseManager` for a description
228 of the inherited *address* and *authkey* optional input arguments and how
229 they may be used to connect to an existing :class:`!SharedMemoryManager` service
230 from other processes.
231
232 .. method:: SharedMemory(size)
233
234 Create and return a new :class:`SharedMemory` object with the
235 specified *size* in bytes.
236
237 .. method:: ShareableList(sequence)
238
239 Create and return a new :class:`ShareableList` object, initialized
240 by the values from the input *sequence*.
241
242
243 The following example demonstrates the basic mechanisms of a
244 :class:`~multiprocessing.managers.SharedMemoryManager`:
245
246 .. doctest::
247 :options: +SKIP
248
249 >>> from multiprocessing.managers import SharedMemoryManager
250 >>> smm = SharedMemoryManager()
251 >>> smm.start() # Start the process that manages the shared memory blocks
252 >>> sl = smm.ShareableList(range(4))
253 >>> sl
254 ShareableList([0, 1, 2, 3], name='psm_6572_7512')
255 >>> raw_shm = smm.SharedMemory(size=128)
256 >>> another_sl = smm.ShareableList('alpha')
257 >>> another_sl
258 ShareableList(['a', 'l', 'p', 'h', 'a'], name='psm_6572_12221')
259 >>> smm.shutdown() # Calls unlink() on sl, raw_shm, and another_sl
260
261 The following example depicts a potentially more convenient pattern for using
262 :class:`~multiprocessing.managers.SharedMemoryManager` objects via the
263 :keyword:`with` statement to ensure that all shared memory blocks are released
264 after they are no longer needed:
265
266 .. doctest::
267 :options: +SKIP
268
269 >>> with SharedMemoryManager() as smm:
270 ... sl = smm.ShareableList(range(2000))
271 ... # Divide the work among two processes, storing partial results in sl
272 ... p1 = Process(target=do_work, args=(sl, 0, 1000))
273 ... p2 = Process(target=do_work, args=(sl, 1000, 2000))
274 ... p1.start()
275 ... p2.start() # A multiprocessing.Pool might be more efficient
276 ... p1.join()
277 ... p2.join() # Wait for all work to complete in both processes
278 ... total_result = sum(sl) # Consolidate the partial results now in sl
279
280 When using a :class:`~multiprocessing.managers.SharedMemoryManager`
281 in a :keyword:`with` statement, the shared memory blocks created using that
282 manager are all released when the :keyword:`!with` statement's code block
283 finishes execution.
284
285
286 .. class:: ShareableList(sequence=None, *, name=None)
287
288 Provide a mutable list-like object where all values stored within are
289 stored in a shared memory block.
290 This constrains storable values to the following built-in data types:
291
292 * :class:`int` (signed 64-bit)
293 * :class:`float`
294 * :class:`bool`
295 * :class:`str` (less than 10M bytes each when encoded as UTF-8)
296 * :class:`bytes` (less than 10M bytes each)
297 * ``None``
298
299 It also notably differs from the built-in :class:`list` type
300 in that these lists can not change their overall length
301 (i.e. no :meth:`!append`, :meth:`!insert`, etc.) and do not
302 support the dynamic creation of new :class:`!ShareableList` instances
303 via slicing.
304
305 *sequence* is used in populating a new :class:`!ShareableList` full of values.
306 Set to ``None`` to instead attach to an already existing
307 :class:`!ShareableList` by its unique shared memory name.
308
309 *name* is the unique name for the requested shared memory, as described
310 in the definition for :class:`SharedMemory`. When attaching to an
311 existing :class:`!ShareableList`, specify its shared memory block's unique
312 name while leaving *sequence* set to ``None``.
313
314 .. note::
315
316 A known issue exists for :class:`bytes` and :class:`str` values.
317 If they end with ``\x00`` nul bytes or characters, those may be
318 *silently stripped* when fetching them by index from the
319 :class:`!ShareableList`. This ``.rstrip(b'\x00')`` behavior is
320 considered a bug and may go away in the future. See :gh:`106939`.
321
322 For applications where rstripping of trailing nulls is a problem,
323 work around it by always unconditionally appending an extra non-0
324 byte to the end of such values when storing and unconditionally
325 removing it when fetching:
326
327 .. doctest::
328
329 >>> from multiprocessing import shared_memory
330 >>> nul_bug_demo = shared_memory.ShareableList(['?\x00', b'\x03\x02\x01\x00\x00\x00'])
331 >>> nul_bug_demo[0]
332 '?'
333 >>> nul_bug_demo[1]
334 b'\x03\x02\x01'
335 >>> nul_bug_demo.shm.unlink()
336 >>> padded = shared_memory.ShareableList(['?\x00\x07', b'\x03\x02\x01\x00\x00\x00\x07'])
337 >>> padded[0][:-1]
338 '?\x00'
339 >>> padded[1][:-1]
340 b'\x03\x02\x01\x00\x00\x00'
341 >>> padded.shm.unlink()
342
343 .. method:: count(value)
344
345 Return the number of occurrences of *value*.
346
347 .. method:: index(value)
348
349 Return first index position of *value*.
350 Raise :exc:`ValueError` if *value* is not present.
351
352 .. attribute:: format
353
354 Read-only attribute containing the :mod:`struct` packing format used by
355 all currently stored values.
356
357 .. attribute:: shm
358
359 The :class:`SharedMemory` instance where the values are stored.
360
361
362 The following example demonstrates basic use of a :class:`ShareableList`
363 instance:
364
365 >>> from multiprocessing import shared_memory
366 >>> a = shared_memory.ShareableList(['howdy', b'HoWdY', -273.154, 100, None, True, 42])
367 >>> [ type(entry) for entry in a ]
368 [<class 'str'>, <class 'bytes'>, <class 'float'>, <class 'int'>, <class 'NoneType'>, <class 'bool'>, <class 'int'>]
369 >>> a[2]
370 -273.154
371 >>> a[2] = -78.5
372 >>> a[2]
373 -78.5
374 >>> a[2] = 'dry ice' # Changing data types is supported as well
375 >>> a[2]
376 'dry ice'
377 >>> a[2] = 'larger than previously allocated storage space'
378 Traceback (most recent call last):
379 ...
380 ValueError: exceeds available storage for existing str
381 >>> a[2]
382 'dry ice'
383 >>> len(a)
384 7
385 >>> a.index(42)
386 6
387 >>> a.count(b'howdy')
388 0
389 >>> a.count(b'HoWdY')
390 1
391 >>> a.shm.close()
392 >>> a.shm.unlink()
393 >>> del a # Use of a ShareableList after call to unlink() is unsupported
394
395 The following example depicts how one, two, or many processes may access the
396 same :class:`ShareableList` by supplying the name of the shared memory block
397 behind it:
398
399 >>> b = shared_memory.ShareableList(range(5)) # In a first process
400 >>> c = shared_memory.ShareableList(name=b.shm.name) # In a second process
401 >>> c
402 ShareableList([0, 1, 2, 3, 4], name='...')
403 >>> c[-1] = -999
404 >>> b[-1]
405 -999
406 >>> b.shm.close()
407 >>> c.shm.close()
408 >>> c.shm.unlink()
409
410 The following examples demonstrates that :class:`ShareableList`
411 (and underlying :class:`SharedMemory`) objects
412 can be pickled and unpickled if needed.
413 Note, that it will still be the same shared object.
414 This happens, because the deserialized object has
415 the same unique name and is just attached to an existing
416 object with the same name (if the object is still alive):
417
418 >>> import pickle
419 >>> from multiprocessing import shared_memory
420 >>> sl = shared_memory.ShareableList(range(10))
421 >>> list(sl)
422 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
423
424 >>> deserialized_sl = pickle.loads(pickle.dumps(sl))
425 >>> list(deserialized_sl)
426 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
427
428 >>> sl[0] = -1
429 >>> deserialized_sl[1] = -2
430 >>> list(sl)
431 [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]
432 >>> list(deserialized_sl)
433 [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]
434
435 >>> sl.shm.close()
436 >>> sl.shm.unlink()