| 1 |
:mod:`!multiprocessing.shared_memory` --- Shared memory for direct access across processes |
| 2 |
========================================================================================== |
| 3 |
|
| 4 |
.. module:: multiprocessing.shared_memory |
| 5 |
:synopsis: Provides shared memory for direct access across processes. |
| 6 |
|
| 7 |
**Source code:** :source:`Lib/multiprocessing/shared_memory.py` |
| 8 |
|
| 9 |
.. versionadded:: 3.8 |
| 10 |
|
| 11 |
.. index:: |
| 12 |
single: Shared Memory |
| 13 |
single: POSIX Shared Memory |
| 14 |
single: Named Shared Memory |
| 15 |
|
| 16 |
-------------- |
| 17 |
|
| 18 |
This module provides a class, :class:`SharedMemory`, for the allocation |
| 19 |
and management of shared memory to be accessed by one or more processes |
| 20 |
on a multicore or symmetric multiprocessor (SMP) machine. To assist with |
| 21 |
the life-cycle management of shared memory especially across distinct |
| 22 |
processes, a :class:`~multiprocessing.managers.BaseManager` subclass, |
| 23 |
:class:`~multiprocessing.managers.SharedMemoryManager`, is also provided in the |
| 24 |
:mod:`multiprocessing.managers` module. |
| 25 |
|
| 26 |
In this module, shared memory refers to "POSIX style" shared memory blocks |
| 27 |
(though is not necessarily implemented explicitly as such) and does not refer |
| 28 |
to "distributed shared memory". This style of shared memory permits distinct |
| 29 |
processes to potentially read and write to a common (or shared) region of |
| 30 |
volatile memory. Processes are conventionally limited to only have access to |
| 31 |
their own process memory space but shared memory permits the sharing |
| 32 |
of data between processes, avoiding the need to instead send messages between |
| 33 |
processes containing that data. Sharing data directly via memory can provide |
| 34 |
significant performance benefits compared to sharing data via disk or socket |
| 35 |
or other communications requiring the serialization/deserialization and |
| 36 |
copying of data. |
| 37 |
|
| 38 |
|
| 39 |
.. class:: SharedMemory(name=None, create=False, size=0, *, track=True) |
| 40 |
|
| 41 |
Create an instance of the :class:`!SharedMemory` class for either |
| 42 |
creating a new shared memory block or attaching to an existing shared |
| 43 |
memory block. Each shared memory block is assigned a unique name. |
| 44 |
In this way, one process can create a shared memory block with a |
| 45 |
particular name and a different process can attach to that same shared |
| 46 |
memory block using that same name. |
| 47 |
|
| 48 |
As a resource for sharing data across processes, shared memory blocks |
| 49 |
may outlive the original process that created them. When one process |
| 50 |
no longer needs access to a shared memory block that might still be |
| 51 |
needed by other processes, the :meth:`close` method should be called. |
| 52 |
When a shared memory block is no longer needed by any process, the |
| 53 |
:meth:`unlink` method should be called to ensure proper cleanup. |
| 54 |
|
| 55 |
:param name: |
| 56 |
The unique name for the requested shared memory, specified as a string. |
| 57 |
When creating a new shared memory block, if ``None`` (the default) |
| 58 |
is supplied for the name, a novel name will be generated. |
| 59 |
:type name: str | None |
| 60 |
|
| 61 |
:param bool create: |
| 62 |
Control whether a new shared memory block is created (``True``) |
| 63 |
or an existing shared memory block is attached (``False``). |
| 64 |
|
| 65 |
:param int size: |
| 66 |
The requested number of bytes when creating a new shared memory block. |
| 67 |
Because some platforms choose to allocate chunks of memory |
| 68 |
based upon that platform's memory page size, the exact size of the shared |
| 69 |
memory block may be larger or equal to the size requested. |
| 70 |
When attaching to an existing shared memory block, |
| 71 |
the *size* parameter is ignored. |
| 72 |
|
| 73 |
:param bool track: |
| 74 |
When ``True``, register the shared memory block with a resource |
| 75 |
tracker process on platforms where the OS does not do this automatically. |
| 76 |
The resource tracker ensures proper cleanup of the shared memory even |
| 77 |
if all other processes with access to the memory exit without doing so. |
| 78 |
Python processes created from a common ancestor using :mod:`multiprocessing` |
| 79 |
facilities share a single resource tracker process, and the lifetime of |
| 80 |
shared memory segments is handled automatically among these processes. |
| 81 |
Python processes created in any other way will receive their own |
| 82 |
resource tracker when accessing shared memory with *track* enabled. |
| 83 |
This will cause the shared memory to be deleted by the resource tracker |
| 84 |
of the first process that terminates. |
| 85 |
To avoid this issue, users of :mod:`subprocess` or standalone Python |
| 86 |
processes should set *track* to ``False`` when there is already another |
| 87 |
process in place that does the bookkeeping. |
| 88 |
*track* is ignored on Windows, which has its own tracking and |
| 89 |
automatically deletes shared memory when all handles to it have been closed. |
| 90 |
|
| 91 |
.. versionchanged:: 3.13 |
| 92 |
Added the *track* parameter. |
| 93 |
|
| 94 |
.. method:: close() |
| 95 |
|
| 96 |
Close the file descriptor/handle to the shared memory from this |
| 97 |
instance. :meth:`close` should be called once access to the shared |
| 98 |
memory block from this instance is no longer needed. Depending |
| 99 |
on operating system, the underlying memory may or may not be freed |
| 100 |
even if all handles to it have been closed. To ensure proper cleanup, |
| 101 |
use the :meth:`unlink` method. |
| 102 |
|
| 103 |
.. method:: unlink() |
| 104 |
|
| 105 |
Delete the underlying shared memory block. This should be called only |
| 106 |
once per shared memory block regardless of the number of handles to it, |
| 107 |
even in other processes. |
| 108 |
:meth:`unlink` and :meth:`close` can be called in any order, but |
| 109 |
trying to access data inside a shared memory block after :meth:`unlink` |
| 110 |
may result in memory access errors, depending on platform. |
| 111 |
|
| 112 |
This method has no effect on Windows, where the only way to delete a |
| 113 |
shared memory block is to close all handles. |
| 114 |
|
| 115 |
.. attribute:: buf |
| 116 |
|
| 117 |
A memoryview of contents of the shared memory block. |
| 118 |
|
| 119 |
.. attribute:: name |
| 120 |
|
| 121 |
Read-only access to the unique name of the shared memory block. |
| 122 |
|
| 123 |
.. attribute:: size |
| 124 |
|
| 125 |
Read-only access to size in bytes of the shared memory block. |
| 126 |
|
| 127 |
|
| 128 |
The following example demonstrates low-level use of :class:`SharedMemory` |
| 129 |
instances:: |
| 130 |
|
| 131 |
>>> from multiprocessing import shared_memory |
| 132 |
>>> shm_a = shared_memory.SharedMemory(create=True, size=10) |
| 133 |
>>> type(shm_a.buf) |
| 134 |
<class 'memoryview'> |
| 135 |
>>> buffer = shm_a.buf |
| 136 |
>>> len(buffer) |
| 137 |
10 |
| 138 |
>>> buffer[:4] = bytearray([22, 33, 44, 55]) # Modify multiple at once |
| 139 |
>>> buffer[4] = 100 # Modify single byte at a time |
| 140 |
>>> # Attach to an existing shared memory block |
| 141 |
>>> shm_b = shared_memory.SharedMemory(shm_a.name) |
| 142 |
>>> import array |
| 143 |
>>> array.array('b', shm_b.buf[:5]) # Copy the data into a new array.array |
| 144 |
array('b', [22, 33, 44, 55, 100]) |
| 145 |
>>> shm_b.buf[:5] = b'howdy' # Modify via shm_b using bytes |
| 146 |
>>> bytes(shm_a.buf[:5]) # Access via shm_a |
| 147 |
b'howdy' |
| 148 |
>>> shm_b.close() # Close each SharedMemory instance |
| 149 |
>>> shm_a.close() |
| 150 |
>>> shm_a.unlink() # Call unlink only once to release the shared memory |
| 151 |
|
| 152 |
|
| 153 |
|
| 154 |
The following example demonstrates a practical use of the :class:`SharedMemory` |
| 155 |
class with `NumPy arrays <https://numpy.org/>`_, accessing the |
| 156 |
same :class:`!numpy.ndarray` from two distinct Python shells: |
| 157 |
|
| 158 |
.. doctest:: |
| 159 |
:options: +SKIP |
| 160 |
|
| 161 |
>>> # In the first Python interactive shell |
| 162 |
>>> import numpy as np |
| 163 |
>>> a = np.array([1, 1, 2, 3, 5, 8]) # Start with an existing NumPy array |
| 164 |
>>> from multiprocessing import shared_memory |
| 165 |
>>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes) |
| 166 |
>>> # Now create a NumPy array backed by shared memory |
| 167 |
>>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf) |
| 168 |
>>> b[:] = a[:] # Copy the original data into shared memory |
| 169 |
>>> b |
| 170 |
array([1, 1, 2, 3, 5, 8]) |
| 171 |
>>> type(b) |
| 172 |
<class 'numpy.ndarray'> |
| 173 |
>>> type(a) |
| 174 |
<class 'numpy.ndarray'> |
| 175 |
>>> shm.name # We did not specify a name so one was chosen for us |
| 176 |
'psm_21467_46075' |
| 177 |
|
| 178 |
>>> # In either the same shell or a new Python shell on the same machine |
| 179 |
>>> import numpy as np |
| 180 |
>>> from multiprocessing import shared_memory |
| 181 |
>>> # Attach to the existing shared memory block |
| 182 |
>>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075') |
| 183 |
>>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example |
| 184 |
>>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf) |
| 185 |
>>> c |
| 186 |
array([1, 1, 2, 3, 5, 8]) |
| 187 |
>>> c[-1] = 888 |
| 188 |
>>> c |
| 189 |
array([ 1, 1, 2, 3, 5, 888]) |
| 190 |
|
| 191 |
>>> # Back in the first Python interactive shell, b reflects this change |
| 192 |
>>> b |
| 193 |
array([ 1, 1, 2, 3, 5, 888]) |
| 194 |
|
| 195 |
>>> # Clean up from within the second Python shell |
| 196 |
>>> del c # Unnecessary; merely emphasizing the array is no longer used |
| 197 |
>>> existing_shm.close() |
| 198 |
|
| 199 |
>>> # Clean up from within the first Python shell |
| 200 |
>>> del b # Unnecessary; merely emphasizing the array is no longer used |
| 201 |
>>> shm.close() |
| 202 |
>>> shm.unlink() # Free and release the shared memory block at the very end |
| 203 |
|
| 204 |
|
| 205 |
.. class:: SharedMemoryManager([address[, authkey]]) |
| 206 |
:module: multiprocessing.managers |
| 207 |
|
| 208 |
A subclass of :class:`multiprocessing.managers.BaseManager` which can be |
| 209 |
used for the management of shared memory blocks across processes. |
| 210 |
|
| 211 |
A call to :meth:`~multiprocessing.managers.BaseManager.start` on a |
| 212 |
:class:`!SharedMemoryManager` instance causes a new process to be started. |
| 213 |
This new process's sole purpose is to manage the life cycle |
| 214 |
of all shared memory blocks created through it. To trigger the release |
| 215 |
of all shared memory blocks managed by that process, call |
| 216 |
:meth:`~multiprocessing.managers.BaseManager.shutdown` on the instance. |
| 217 |
This triggers a :meth:`~multiprocessing.shared_memory.SharedMemory.unlink` call |
| 218 |
on all of the :class:`SharedMemory` objects managed by that process and then |
| 219 |
stops the process itself. By creating :class:`!SharedMemory` instances |
| 220 |
through a :class:`!SharedMemoryManager`, we avoid the need to manually track |
| 221 |
and trigger the freeing of shared memory resources. |
| 222 |
|
| 223 |
This class provides methods for creating and returning :class:`SharedMemory` |
| 224 |
instances and for creating a list-like object (:class:`ShareableList`) |
| 225 |
backed by shared memory. |
| 226 |
|
| 227 |
Refer to :class:`~multiprocessing.managers.BaseManager` for a description |
| 228 |
of the inherited *address* and *authkey* optional input arguments and how |
| 229 |
they may be used to connect to an existing :class:`!SharedMemoryManager` service |
| 230 |
from other processes. |
| 231 |
|
| 232 |
.. method:: SharedMemory(size) |
| 233 |
|
| 234 |
Create and return a new :class:`SharedMemory` object with the |
| 235 |
specified *size* in bytes. |
| 236 |
|
| 237 |
.. method:: ShareableList(sequence) |
| 238 |
|
| 239 |
Create and return a new :class:`ShareableList` object, initialized |
| 240 |
by the values from the input *sequence*. |
| 241 |
|
| 242 |
|
| 243 |
The following example demonstrates the basic mechanisms of a |
| 244 |
:class:`~multiprocessing.managers.SharedMemoryManager`: |
| 245 |
|
| 246 |
.. doctest:: |
| 247 |
:options: +SKIP |
| 248 |
|
| 249 |
>>> from multiprocessing.managers import SharedMemoryManager |
| 250 |
>>> smm = SharedMemoryManager() |
| 251 |
>>> smm.start() # Start the process that manages the shared memory blocks |
| 252 |
>>> sl = smm.ShareableList(range(4)) |
| 253 |
>>> sl |
| 254 |
ShareableList([0, 1, 2, 3], name='psm_6572_7512') |
| 255 |
>>> raw_shm = smm.SharedMemory(size=128) |
| 256 |
>>> another_sl = smm.ShareableList('alpha') |
| 257 |
>>> another_sl |
| 258 |
ShareableList(['a', 'l', 'p', 'h', 'a'], name='psm_6572_12221') |
| 259 |
>>> smm.shutdown() # Calls unlink() on sl, raw_shm, and another_sl |
| 260 |
|
| 261 |
The following example depicts a potentially more convenient pattern for using |
| 262 |
:class:`~multiprocessing.managers.SharedMemoryManager` objects via the |
| 263 |
:keyword:`with` statement to ensure that all shared memory blocks are released |
| 264 |
after they are no longer needed: |
| 265 |
|
| 266 |
.. doctest:: |
| 267 |
:options: +SKIP |
| 268 |
|
| 269 |
>>> with SharedMemoryManager() as smm: |
| 270 |
... sl = smm.ShareableList(range(2000)) |
| 271 |
... # Divide the work among two processes, storing partial results in sl |
| 272 |
... p1 = Process(target=do_work, args=(sl, 0, 1000)) |
| 273 |
... p2 = Process(target=do_work, args=(sl, 1000, 2000)) |
| 274 |
... p1.start() |
| 275 |
... p2.start() # A multiprocessing.Pool might be more efficient |
| 276 |
... p1.join() |
| 277 |
... p2.join() # Wait for all work to complete in both processes |
| 278 |
... total_result = sum(sl) # Consolidate the partial results now in sl |
| 279 |
|
| 280 |
When using a :class:`~multiprocessing.managers.SharedMemoryManager` |
| 281 |
in a :keyword:`with` statement, the shared memory blocks created using that |
| 282 |
manager are all released when the :keyword:`!with` statement's code block |
| 283 |
finishes execution. |
| 284 |
|
| 285 |
|
| 286 |
.. class:: ShareableList(sequence=None, *, name=None) |
| 287 |
|
| 288 |
Provide a mutable list-like object where all values stored within are |
| 289 |
stored in a shared memory block. |
| 290 |
This constrains storable values to the following built-in data types: |
| 291 |
|
| 292 |
* :class:`int` (signed 64-bit) |
| 293 |
* :class:`float` |
| 294 |
* :class:`bool` |
| 295 |
* :class:`str` (less than 10M bytes each when encoded as UTF-8) |
| 296 |
* :class:`bytes` (less than 10M bytes each) |
| 297 |
* ``None`` |
| 298 |
|
| 299 |
It also notably differs from the built-in :class:`list` type |
| 300 |
in that these lists can not change their overall length |
| 301 |
(i.e. no :meth:`!append`, :meth:`!insert`, etc.) and do not |
| 302 |
support the dynamic creation of new :class:`!ShareableList` instances |
| 303 |
via slicing. |
| 304 |
|
| 305 |
*sequence* is used in populating a new :class:`!ShareableList` full of values. |
| 306 |
Set to ``None`` to instead attach to an already existing |
| 307 |
:class:`!ShareableList` by its unique shared memory name. |
| 308 |
|
| 309 |
*name* is the unique name for the requested shared memory, as described |
| 310 |
in the definition for :class:`SharedMemory`. When attaching to an |
| 311 |
existing :class:`!ShareableList`, specify its shared memory block's unique |
| 312 |
name while leaving *sequence* set to ``None``. |
| 313 |
|
| 314 |
.. note:: |
| 315 |
|
| 316 |
A known issue exists for :class:`bytes` and :class:`str` values. |
| 317 |
If they end with ``\x00`` nul bytes or characters, those may be |
| 318 |
*silently stripped* when fetching them by index from the |
| 319 |
:class:`!ShareableList`. This ``.rstrip(b'\x00')`` behavior is |
| 320 |
considered a bug and may go away in the future. See :gh:`106939`. |
| 321 |
|
| 322 |
For applications where rstripping of trailing nulls is a problem, |
| 323 |
work around it by always unconditionally appending an extra non-0 |
| 324 |
byte to the end of such values when storing and unconditionally |
| 325 |
removing it when fetching: |
| 326 |
|
| 327 |
.. doctest:: |
| 328 |
|
| 329 |
>>> from multiprocessing import shared_memory |
| 330 |
>>> nul_bug_demo = shared_memory.ShareableList(['?\x00', b'\x03\x02\x01\x00\x00\x00']) |
| 331 |
>>> nul_bug_demo[0] |
| 332 |
'?' |
| 333 |
>>> nul_bug_demo[1] |
| 334 |
b'\x03\x02\x01' |
| 335 |
>>> nul_bug_demo.shm.unlink() |
| 336 |
>>> padded = shared_memory.ShareableList(['?\x00\x07', b'\x03\x02\x01\x00\x00\x00\x07']) |
| 337 |
>>> padded[0][:-1] |
| 338 |
'?\x00' |
| 339 |
>>> padded[1][:-1] |
| 340 |
b'\x03\x02\x01\x00\x00\x00' |
| 341 |
>>> padded.shm.unlink() |
| 342 |
|
| 343 |
.. method:: count(value) |
| 344 |
|
| 345 |
Return the number of occurrences of *value*. |
| 346 |
|
| 347 |
.. method:: index(value) |
| 348 |
|
| 349 |
Return first index position of *value*. |
| 350 |
Raise :exc:`ValueError` if *value* is not present. |
| 351 |
|
| 352 |
.. attribute:: format |
| 353 |
|
| 354 |
Read-only attribute containing the :mod:`struct` packing format used by |
| 355 |
all currently stored values. |
| 356 |
|
| 357 |
.. attribute:: shm |
| 358 |
|
| 359 |
The :class:`SharedMemory` instance where the values are stored. |
| 360 |
|
| 361 |
|
| 362 |
The following example demonstrates basic use of a :class:`ShareableList` |
| 363 |
instance: |
| 364 |
|
| 365 |
>>> from multiprocessing import shared_memory |
| 366 |
>>> a = shared_memory.ShareableList(['howdy', b'HoWdY', -273.154, 100, None, True, 42]) |
| 367 |
>>> [ type(entry) for entry in a ] |
| 368 |
[<class 'str'>, <class 'bytes'>, <class 'float'>, <class 'int'>, <class 'NoneType'>, <class 'bool'>, <class 'int'>] |
| 369 |
>>> a[2] |
| 370 |
-273.154 |
| 371 |
>>> a[2] = -78.5 |
| 372 |
>>> a[2] |
| 373 |
-78.5 |
| 374 |
>>> a[2] = 'dry ice' # Changing data types is supported as well |
| 375 |
>>> a[2] |
| 376 |
'dry ice' |
| 377 |
>>> a[2] = 'larger than previously allocated storage space' |
| 378 |
Traceback (most recent call last): |
| 379 |
... |
| 380 |
ValueError: exceeds available storage for existing str |
| 381 |
>>> a[2] |
| 382 |
'dry ice' |
| 383 |
>>> len(a) |
| 384 |
7 |
| 385 |
>>> a.index(42) |
| 386 |
6 |
| 387 |
>>> a.count(b'howdy') |
| 388 |
0 |
| 389 |
>>> a.count(b'HoWdY') |
| 390 |
1 |
| 391 |
>>> a.shm.close() |
| 392 |
>>> a.shm.unlink() |
| 393 |
>>> del a # Use of a ShareableList after call to unlink() is unsupported |
| 394 |
|
| 395 |
The following example depicts how one, two, or many processes may access the |
| 396 |
same :class:`ShareableList` by supplying the name of the shared memory block |
| 397 |
behind it: |
| 398 |
|
| 399 |
>>> b = shared_memory.ShareableList(range(5)) # In a first process |
| 400 |
>>> c = shared_memory.ShareableList(name=b.shm.name) # In a second process |
| 401 |
>>> c |
| 402 |
ShareableList([0, 1, 2, 3, 4], name='...') |
| 403 |
>>> c[-1] = -999 |
| 404 |
>>> b[-1] |
| 405 |
-999 |
| 406 |
>>> b.shm.close() |
| 407 |
>>> c.shm.close() |
| 408 |
>>> c.shm.unlink() |
| 409 |
|
| 410 |
The following examples demonstrates that :class:`ShareableList` |
| 411 |
(and underlying :class:`SharedMemory`) objects |
| 412 |
can be pickled and unpickled if needed. |
| 413 |
Note, that it will still be the same shared object. |
| 414 |
This happens, because the deserialized object has |
| 415 |
the same unique name and is just attached to an existing |
| 416 |
object with the same name (if the object is still alive): |
| 417 |
|
| 418 |
>>> import pickle |
| 419 |
>>> from multiprocessing import shared_memory |
| 420 |
>>> sl = shared_memory.ShareableList(range(10)) |
| 421 |
>>> list(sl) |
| 422 |
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |
| 423 |
|
| 424 |
>>> deserialized_sl = pickle.loads(pickle.dumps(sl)) |
| 425 |
>>> list(deserialized_sl) |
| 426 |
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |
| 427 |
|
| 428 |
>>> sl[0] = -1 |
| 429 |
>>> deserialized_sl[1] = -2 |
| 430 |
>>> list(sl) |
| 431 |
[-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] |
| 432 |
>>> list(deserialized_sl) |
| 433 |
[-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] |
| 434 |
|
| 435 |
>>> sl.shm.close() |
| 436 |
>>> sl.shm.unlink() |