Debian Codegraph @

Doc/library/mmap.rst

raw
1 :mod:`!mmap` --- Memory-mapped file support
2 ===========================================
3
4 .. module:: mmap
5 :synopsis: Interface to memory-mapped files for Unix and Windows.
6
7 --------------
8
9 .. include:: ../includes/wasm-notavail.rst
10
11 Memory-mapped file objects behave like both :class:`bytearray` and like
12 :term:`file objects <file object>`. You can use mmap objects in most places
13 where :class:`bytearray` are expected; for example, you can use the :mod:`re`
14 module to search through a memory-mapped file. You can also change a single
15 byte by doing ``obj[index] = 97``, or change a subsequence by assigning to a
16 slice: ``obj[i1:i2] = b'...'``. You can also read and write data starting at
17 the current file position, and :meth:`seek` through the file to different positions.
18
19 A memory-mapped file is created by the :class:`~mmap.mmap` constructor, which is
20 different on Unix and on Windows. In either case you must provide a file
21 descriptor for a file opened for update. If you wish to map an existing Python
22 file object, use its :meth:`~io.IOBase.fileno` method to obtain the correct value for the
23 *fileno* parameter. Otherwise, you can open the file using the
24 :func:`os.open` function, which returns a file descriptor directly (the file
25 still needs to be closed when done).
26
27 .. note::
28 If you want to create a memory-mapping for a writable, buffered file, you
29 should :func:`~io.IOBase.flush` the file first. This is necessary to ensure
30 that local modifications to the buffers are actually available to the
31 mapping.
32
33 For both the Unix and Windows versions of the constructor, *access* may be
34 specified as an optional keyword parameter. *access* accepts one of four
35 values: :const:`ACCESS_READ`, :const:`ACCESS_WRITE`, or :const:`ACCESS_COPY` to
36 specify read-only, write-through or copy-on-write memory respectively, or
37 :const:`ACCESS_DEFAULT` to defer to *prot*. *access* can be used on both Unix
38 and Windows. If *access* is not specified, Windows mmap returns a
39 write-through mapping. The initial memory values for all three access types
40 are taken from the specified file. Assignment to an :const:`ACCESS_READ`
41 memory map raises a :exc:`TypeError` exception. Assignment to an
42 :const:`ACCESS_WRITE` memory map affects both memory and the underlying file.
43 Assignment to an :const:`ACCESS_COPY` memory map affects memory but does not
44 update the underlying file.
45
46 .. versionchanged:: 3.7
47 Added :const:`ACCESS_DEFAULT` constant.
48
49 To map anonymous memory, -1 should be passed as the fileno along with the length.
50
51 .. class:: mmap(fileno, length, tagname=None, access=ACCESS_DEFAULT, offset=0)
52
53 **(Windows version)** Maps *length* bytes from the file specified by the
54 file handle *fileno*, and creates a mmap object. If *length* is larger
55 than the current size of the file, the file is extended to contain *length*
56 bytes. If *length* is ``0``, the maximum length of the map is the current
57 size of the file, except that if the file is empty Windows raises an
58 exception (you cannot create an empty mapping on Windows).
59
60 *tagname*, if specified and not ``None``, is a string giving a tag name for
61 the mapping. Windows allows you to have many different mappings against
62 the same file. If you specify the name of an existing tag, that tag is
63 opened, otherwise a new tag of this name is created. If this parameter is
64 omitted or ``None``, the mapping is created without a name. Avoiding the
65 use of the *tagname* parameter will assist in keeping your code portable
66 between Unix and Windows.
67
68 *offset* may be specified as a non-negative integer offset. mmap references
69 will be relative to the offset from the beginning of the file. *offset*
70 defaults to 0. *offset* must be a multiple of the :const:`ALLOCATIONGRANULARITY`.
71
72 .. audit-event:: mmap.__new__ fileno,length,access,offset mmap.mmap
73
74 .. class:: mmap(fileno, length, flags=MAP_SHARED, prot=PROT_WRITE|PROT_READ, \
75 access=ACCESS_DEFAULT, offset=0, *, trackfd=True)
76 :noindex:
77
78 **(Unix version)** Maps *length* bytes from the file specified by the file
79 descriptor *fileno*, and returns a mmap object. If *length* is ``0``, the
80 maximum length of the map will be the current size of the file when
81 :class:`~mmap.mmap` is called.
82
83 *flags* specifies the nature of the mapping. :const:`MAP_PRIVATE` creates a
84 private copy-on-write mapping, so changes to the contents of the mmap
85 object will be private to this process, and :const:`MAP_SHARED` creates a
86 mapping that's shared with all other processes mapping the same areas of
87 the file. The default value is :const:`MAP_SHARED`. Some systems have
88 additional possible flags with the full list specified in
89 :ref:`MAP_* constants <map-constants>`.
90
91 *prot*, if specified, gives the desired memory protection; the two most
92 useful values are :const:`PROT_READ` and :const:`PROT_WRITE`, to specify
93 that the pages may be read or written. *prot* defaults to
94 :const:`PROT_READ \| PROT_WRITE`.
95
96 *access* may be specified in lieu of *flags* and *prot* as an optional
97 keyword parameter. It is an error to specify both *flags*, *prot* and
98 *access*. See the description of *access* above for information on how to
99 use this parameter.
100
101 *offset* may be specified as a non-negative integer offset. mmap references
102 will be relative to the offset from the beginning of the file. *offset*
103 defaults to 0. *offset* must be a multiple of :const:`ALLOCATIONGRANULARITY`
104 which is equal to :const:`PAGESIZE` on Unix systems.
105
106 If *trackfd* is ``False``, the file descriptor specified by *fileno* will
107 not be duplicated, and the resulting :class:`!mmap` object will not
108 be associated with the map's underlying file.
109 This means that the :meth:`~mmap.mmap.size` and :meth:`~mmap.mmap.resize`
110 methods will fail.
111 This mode is useful to limit the number of open file descriptors.
112
113 To ensure validity of the created memory mapping the file specified
114 by the descriptor *fileno* is internally automatically synchronized
115 with the physical backing store on macOS.
116
117 .. versionchanged:: 3.13
118 The *trackfd* parameter was added.
119
120 This example shows a simple way of using :class:`~mmap.mmap`::
121
122 import mmap
123
124 # write a simple example file
125 with open("hello.txt", "wb") as f:
126 f.write(b"Hello Python!\n")
127
128 with open("hello.txt", "r+b") as f:
129 # memory-map the file, size 0 means whole file
130 mm = mmap.mmap(f.fileno(), 0)
131 # read content via standard file methods
132 print(mm.readline()) # prints b"Hello Python!\n"
133 # read content via slice notation
134 print(mm[:5]) # prints b"Hello"
135 # update content using slice notation;
136 # note that new content must have same size
137 mm[6:] = b" world!\n"
138 # ... and read again using standard file methods
139 mm.seek(0)
140 print(mm.readline()) # prints b"Hello world!\n"
141 # close the map
142 mm.close()
143
144
145 :class:`~mmap.mmap` can also be used as a context manager in a :keyword:`with`
146 statement::
147
148 import mmap
149
150 with mmap.mmap(-1, 13) as mm:
151 mm.write(b"Hello world!")
152
153 .. versionadded:: 3.2
154 Context manager support.
155
156
157 The next example demonstrates how to create an anonymous map and exchange
158 data between the parent and child processes::
159
160 import mmap
161 import os
162
163 mm = mmap.mmap(-1, 13)
164 mm.write(b"Hello world!")
165
166 pid = os.fork()
167
168 if pid == 0: # In a child process
169 mm.seek(0)
170 print(mm.readline())
171
172 mm.close()
173
174 .. audit-event:: mmap.__new__ fileno,length,access,offset mmap.mmap
175
176 Memory-mapped file objects support the following methods:
177
178 .. method:: close()
179
180 Closes the mmap. Subsequent calls to other methods of the object will
181 result in a ValueError exception being raised. This will not close
182 the open file.
183
184
185 .. attribute:: closed
186
187 ``True`` if the file is closed.
188
189 .. versionadded:: 3.2
190
191
192 .. method:: find(sub[, start[, end]])
193
194 Returns the lowest index in the object where the subsequence *sub* is
195 found, such that *sub* is contained in the range [*start*, *end*].
196 Optional arguments *start* and *end* are interpreted as in slice notation.
197 Returns ``-1`` on failure.
198
199 .. versionchanged:: 3.5
200 Writable :term:`bytes-like object` is now accepted.
201
202
203 .. method:: flush()
204 flush(offset, size, /)
205
206 Flushes changes made to the in-memory copy of a file back to disk. Without
207 use of this call there is no guarantee that changes are written back before
208 the object is destroyed. If *offset* and *size* are specified, only
209 changes to the given range of bytes will be flushed to disk; otherwise, the
210 whole extent of the mapping is flushed. *offset* must be a multiple of the
211 :const:`PAGESIZE` or :const:`ALLOCATIONGRANULARITY`.
212
213 ``None`` is returned to indicate success. An exception is raised when the
214 call failed.
215
216 .. versionchanged:: 3.8
217 Previously, a nonzero value was returned on success; zero was returned
218 on error under Windows. A zero value was returned on success; an
219 exception was raised on error under Unix.
220
221
222 .. method:: madvise(option[, start[, length]])
223
224 Send advice *option* to the kernel about the memory region beginning at
225 *start* and extending *length* bytes. *option* must be one of the
226 :ref:`MADV_* constants <madvise-constants>` available on the system. If
227 *start* and *length* are omitted, the entire mapping is spanned. On
228 some systems (including Linux), *start* must be a multiple of the
229 :const:`PAGESIZE`.
230
231 Availability: Systems with the ``madvise()`` system call.
232
233 .. versionadded:: 3.8
234
235
236 .. method:: move(dest, src, count)
237
238 Copy the *count* bytes starting at offset *src* to the destination index
239 *dest*. If the mmap was created with :const:`ACCESS_READ`, then calls to
240 move will raise a :exc:`TypeError` exception.
241
242
243 .. method:: read([n])
244
245 Return a :class:`bytes` containing up to *n* bytes starting from the
246 current file position. If the argument is omitted, ``None`` or negative,
247 return all bytes from the current file position to the end of the
248 mapping. The file position is updated to point after the bytes that were
249 returned.
250
251 .. versionchanged:: 3.3
252 Argument can be omitted or ``None``.
253
254 .. method:: read_byte()
255
256 Returns a byte at the current file position as an integer, and advances
257 the file position by 1.
258
259
260 .. method:: readline()
261
262 Returns a single line, starting at the current file position and up to the
263 next newline. The file position is updated to point after the bytes that were
264 returned.
265
266
267 .. method:: resize(newsize)
268
269 Resizes the map and the underlying file, if any.
270
271 Resizing a map created with *access* of :const:`ACCESS_READ` or
272 :const:`ACCESS_COPY`, will raise a :exc:`TypeError` exception.
273 Resizing a map created with *trackfd* set to ``False``,
274 will raise a :exc:`ValueError` exception.
275
276 **On Windows**: Resizing the map will raise an :exc:`OSError` if there are other
277 maps against the same named file. Resizing an anonymous map (ie against the
278 pagefile) will silently create a new map with the original data copied over
279 up to the length of the new size.
280
281 .. versionchanged:: 3.11
282 Correctly fails if attempting to resize when another map is held
283 Allows resize against an anonymous map on Windows
284
285 .. method:: rfind(sub[, start[, end]])
286
287 Returns the highest index in the object where the subsequence *sub* is
288 found, such that *sub* is contained in the range [*start*, *end*].
289 Optional arguments *start* and *end* are interpreted as in slice notation.
290 Returns ``-1`` on failure.
291
292 .. versionchanged:: 3.5
293 Writable :term:`bytes-like object` is now accepted.
294
295
296 .. method:: seek(pos[, whence])
297
298 Set the file's current position. *whence* argument is optional and
299 defaults to ``os.SEEK_SET`` or ``0`` (absolute file positioning); other
300 values are ``os.SEEK_CUR`` or ``1`` (seek relative to the current
301 position) and ``os.SEEK_END`` or ``2`` (seek relative to the file's end).
302
303 .. versionchanged:: 3.13
304 Return the new absolute position instead of ``None``.
305
306 .. method:: seekable()
307
308 Return whether the file supports seeking, and the return value is always ``True``.
309
310 .. versionadded:: 3.13
311
312 .. method:: size()
313
314 Return the length of the file, which can be larger than the size of the
315 memory-mapped area.
316
317
318 .. method:: tell()
319
320 Returns the current position of the file pointer.
321
322
323 .. method:: write(bytes)
324
325 Write the bytes in *bytes* into memory at the current position of the
326 file pointer and return the number of bytes written (never less than
327 ``len(bytes)``, since if the write fails, a :exc:`ValueError` will be
328 raised). The file position is updated to point after the bytes that
329 were written. If the mmap was created with :const:`ACCESS_READ`, then
330 writing to it will raise a :exc:`TypeError` exception.
331
332 .. versionchanged:: 3.5
333 Writable :term:`bytes-like object` is now accepted.
334
335 .. versionchanged:: 3.6
336 The number of bytes written is now returned.
337
338
339 .. method:: write_byte(byte)
340
341 Write the integer *byte* into memory at the current
342 position of the file pointer; the file position is advanced by ``1``. If
343 the mmap was created with :const:`ACCESS_READ`, then writing to it will
344 raise a :exc:`TypeError` exception.
345
346 .. _madvise-constants:
347
348 MADV_* Constants
349 ++++++++++++++++
350
351 .. data:: MADV_NORMAL
352 MADV_RANDOM
353 MADV_SEQUENTIAL
354 MADV_WILLNEED
355 MADV_DONTNEED
356 MADV_REMOVE
357 MADV_DONTFORK
358 MADV_DOFORK
359 MADV_HWPOISON
360 MADV_MERGEABLE
361 MADV_UNMERGEABLE
362 MADV_SOFT_OFFLINE
363 MADV_HUGEPAGE
364 MADV_NOHUGEPAGE
365 MADV_DONTDUMP
366 MADV_DODUMP
367 MADV_FREE
368 MADV_NOSYNC
369 MADV_AUTOSYNC
370 MADV_NOCORE
371 MADV_CORE
372 MADV_PROTECT
373 MADV_FREE_REUSABLE
374 MADV_FREE_REUSE
375
376 These options can be passed to :meth:`mmap.madvise`. Not every option will
377 be present on every system.
378
379 Availability: Systems with the madvise() system call.
380
381 .. versionadded:: 3.8
382
383 .. _map-constants:
384
385 MAP_* Constants
386 +++++++++++++++
387
388 .. data:: MAP_SHARED
389 MAP_PRIVATE
390 MAP_32BIT
391 MAP_ALIGNED_SUPER
392 MAP_ANON
393 MAP_ANONYMOUS
394 MAP_CONCEAL
395 MAP_DENYWRITE
396 MAP_EXECUTABLE
397 MAP_HASSEMAPHORE
398 MAP_JIT
399 MAP_NOCACHE
400 MAP_NOEXTEND
401 MAP_NORESERVE
402 MAP_POPULATE
403 MAP_RESILIENT_CODESIGN
404 MAP_RESILIENT_MEDIA
405 MAP_STACK
406 MAP_TPRO
407 MAP_TRANSLATED_ALLOW_EXECUTE
408 MAP_UNIX03
409
410 These are the various flags that can be passed to :meth:`mmap.mmap`. :data:`MAP_ALIGNED_SUPER`
411 is only available at FreeBSD and :data:`MAP_CONCEAL` is only available at OpenBSD. Note
412 that some options might not be present on some systems.
413
414 .. versionchanged:: 3.10
415 Added :data:`MAP_POPULATE` constant.
416
417 .. versionadded:: 3.11
418 Added :data:`MAP_STACK` constant.
419
420 .. versionadded:: 3.12
421 Added :data:`MAP_ALIGNED_SUPER` and :data:`MAP_CONCEAL` constants.
422
423 .. versionadded:: 3.13
424 Added :data:`MAP_32BIT`, :data:`MAP_HASSEMAPHORE`, :data:`MAP_JIT`,
425 :data:`MAP_NOCACHE`, :data:`MAP_NOEXTEND`, :data:`MAP_NORESERVE`,
426 :data:`MAP_RESILIENT_CODESIGN`, :data:`MAP_RESILIENT_MEDIA`,
427 :data:`MAP_TPRO`, :data:`MAP_TRANSLATED_ALLOW_EXECUTE`, and
428 :data:`MAP_UNIX03` constants.
429