Debian Codegraph @

Doc/library/io.rst

raw
1 :mod:`!io` --- Core tools for working with streams
2 ==================================================
3
4 .. module:: io
5 :synopsis: Core tools for working with streams.
6
7 .. moduleauthor:: Guido van Rossum <[email protected]>
8 .. moduleauthor:: Mike Verdone <[email protected]>
9 .. moduleauthor:: Mark Russell <[email protected]>
10 .. moduleauthor:: Antoine Pitrou <[email protected]>
11 .. moduleauthor:: Amaury Forgeot d'Arc <[email protected]>
12 .. moduleauthor:: Benjamin Peterson <[email protected]>
13 .. sectionauthor:: Benjamin Peterson <[email protected]>
14
15 **Source code:** :source:`Lib/io.py`
16
17 --------------
18
19 .. _io-overview:
20
21 Overview
22 --------
23
24 .. index::
25 single: file object; io module
26
27 The :mod:`io` module provides Python's main facilities for dealing with various
28 types of I/O. There are three main types of I/O: *text I/O*, *binary I/O*
29 and *raw I/O*. These are generic categories, and various backing stores can
30 be used for each of them. A concrete object belonging to any of these
31 categories is called a :term:`file object`. Other common terms are *stream*
32 and *file-like object*.
33
34 Independent of its category, each concrete stream object will also have
35 various capabilities: it can be read-only, write-only, or read-write. It can
36 also allow arbitrary random access (seeking forwards or backwards to any
37 location), or only sequential access (for example in the case of a socket or
38 pipe).
39
40 All streams are careful about the type of data you give to them. For example
41 giving a :class:`str` object to the :meth:`!write` method of a binary stream
42 will raise a :exc:`TypeError`. So will giving a :class:`bytes` object to the
43 :meth:`!write` method of a text stream.
44
45 .. versionchanged:: 3.3
46 Operations that used to raise :exc:`IOError` now raise :exc:`OSError`, since
47 :exc:`IOError` is now an alias of :exc:`OSError`.
48
49 .. _text-io:
50
51 Text I/O
52 ^^^^^^^^
53
54 Text I/O expects and produces :class:`str` objects. This means that whenever
55 the backing store is natively made of bytes (such as in the case of a file),
56 encoding and decoding of data is made transparently as well as optional
57 translation of platform-specific newline characters.
58
59 The easiest way to create a text stream is with :meth:`open`, optionally
60 specifying an encoding::
61
62 f = open("myfile.txt", "r", encoding="utf-8")
63
64 In-memory text streams are also available as :class:`StringIO` objects::
65
66 f = io.StringIO("some initial text data")
67
68 The text stream API is described in detail in the documentation of
69 :class:`TextIOBase`.
70
71 .. _binary-io:
72
73 Binary I/O
74 ^^^^^^^^^^
75
76 Binary I/O (also called *buffered I/O*) expects
77 :term:`bytes-like objects <bytes-like object>` and produces :class:`bytes`
78 objects. No encoding, decoding, or newline translation is performed. This
79 category of streams can be used for all kinds of non-text data, and also when
80 manual control over the handling of text data is desired.
81
82 The easiest way to create a binary stream is with :meth:`open` with ``'b'`` in
83 the mode string::
84
85 f = open("myfile.jpg", "rb")
86
87 In-memory binary streams are also available as :class:`BytesIO` objects::
88
89 f = io.BytesIO(b"some initial binary data: \x00\x01")
90
91 The binary stream API is described in detail in the docs of
92 :class:`BufferedIOBase`.
93
94 Other library modules may provide additional ways to create text or binary
95 streams. See :meth:`socket.socket.makefile` for example.
96
97
98 Raw I/O
99 ^^^^^^^
100
101 Raw I/O (also called *unbuffered I/O*) is generally used as a low-level
102 building-block for binary and text streams; it is rarely useful to directly
103 manipulate a raw stream from user code. Nevertheless, you can create a raw
104 stream by opening a file in binary mode with buffering disabled::
105
106 f = open("myfile.jpg", "rb", buffering=0)
107
108 The raw stream API is described in detail in the docs of :class:`RawIOBase`.
109
110 .. warning::
111 Raw I/O is a low-level interface and methods generally must have their return
112 values checked and be explicitly retried to ensure an operation completes.
113 For instance :meth:`~RawIOBase.write` returns the number of bytes written
114 which may be less than the number of bytes provided (a partial write).
115 High-level I/O objects like :ref:`binary-io` and :ref:`text-io` implement
116 retry behavior.
117
118 .. _io-text-encoding:
119
120 Text Encoding
121 -------------
122
123 The default encoding of :class:`TextIOWrapper` and :func:`open` is
124 locale-specific (:func:`locale.getencoding`).
125
126 However, many developers forget to specify the encoding when opening text files
127 encoded in UTF-8 (e.g. JSON, TOML, Markdown, etc...) since most Unix
128 platforms use UTF-8 locale by default. This causes bugs because the locale
129 encoding is not UTF-8 for most Windows users. For example::
130
131 # May not work on Windows when non-ASCII characters in the file.
132 with open("README.md") as f:
133 long_description = f.read()
134
135 Accordingly, it is highly recommended that you specify the encoding
136 explicitly when opening text files. If you want to use UTF-8, pass
137 ``encoding="utf-8"``. To use the current locale encoding,
138 ``encoding="locale"`` is supported since Python 3.10.
139
140 .. seealso::
141
142 :ref:`utf8-mode`
143 Python UTF-8 Mode can be used to change the default encoding to
144 UTF-8 from locale-specific encoding.
145
146 :pep:`686`
147 Python 3.15 will make :ref:`utf8-mode` default.
148
149 .. _io-encoding-warning:
150
151 Opt-in EncodingWarning
152 ^^^^^^^^^^^^^^^^^^^^^^
153
154 .. versionadded:: 3.10
155 See :pep:`597` for more details.
156
157 To find where the default locale encoding is used, you can enable
158 the :option:`-X warn_default_encoding <-X>` command line option or set the
159 :envvar:`PYTHONWARNDEFAULTENCODING` environment variable, which will
160 emit an :exc:`EncodingWarning` when the default encoding is used.
161
162 If you are providing an API that uses :func:`open` or
163 :class:`TextIOWrapper` and passes ``encoding=None`` as a parameter, you
164 can use :func:`text_encoding` so that callers of the API will emit an
165 :exc:`EncodingWarning` if they don't pass an ``encoding``. However,
166 please consider using UTF-8 by default (i.e. ``encoding="utf-8"``) for
167 new APIs.
168
169
170 High-level Module Interface
171 ---------------------------
172
173 .. data:: DEFAULT_BUFFER_SIZE
174
175 An int containing the default buffer size used by the module's buffered I/O
176 classes. :func:`open` uses the file's blksize (as obtained by
177 :func:`os.stat`) if possible.
178
179
180 .. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)
181
182 This is an alias for the builtin :func:`open` function.
183
184 .. audit-event:: open path,mode,flags io.open
185
186 This function raises an :ref:`auditing event <auditing>` ``open`` with
187 arguments *path*, *mode* and *flags*. The *mode* and *flags*
188 arguments may have been modified or inferred from the original call.
189
190
191 .. function:: open_code(path)
192
193 Opens the provided file with mode ``'rb'``. This function should be used
194 when the intent is to treat the contents as executable code.
195
196 *path* should be a :class:`str` and an absolute path.
197
198 The behavior of this function may be overridden by an earlier call to the
199 :c:func:`PyFile_SetOpenCodeHook`. However, assuming that *path* is a
200 :class:`str` and an absolute path, ``open_code(path)`` should always behave
201 the same as ``open(path, 'rb')``. Overriding the behavior is intended for
202 additional validation or preprocessing of the file.
203
204 .. versionadded:: 3.8
205
206
207 .. function:: text_encoding(encoding, stacklevel=2, /)
208
209 This is a helper function for callables that use :func:`open` or
210 :class:`TextIOWrapper` and have an ``encoding=None`` parameter.
211
212 This function returns *encoding* if it is not ``None``.
213 Otherwise, it returns ``"locale"`` or ``"utf-8"`` depending on
214 :ref:`UTF-8 Mode <utf8-mode>`.
215
216 This function emits an :class:`EncodingWarning` if
217 :data:`sys.flags.warn_default_encoding <sys.flags>` is true and *encoding*
218 is ``None``. *stacklevel* specifies where the warning is emitted.
219 For example::
220
221 def read_text(path, encoding=None):
222 encoding = io.text_encoding(encoding) # stacklevel=2
223 with open(path, encoding) as f:
224 return f.read()
225
226 In this example, an :class:`EncodingWarning` is emitted for the caller of
227 ``read_text()``.
228
229 See :ref:`io-text-encoding` for more information.
230
231 .. versionadded:: 3.10
232
233 .. versionchanged:: 3.11
234 :func:`text_encoding` returns "utf-8" when UTF-8 mode is enabled and
235 *encoding* is ``None``.
236
237
238 .. exception:: BlockingIOError
239
240 This is a compatibility alias for the builtin :exc:`BlockingIOError`
241 exception.
242
243
244 .. exception:: UnsupportedOperation
245
246 An exception inheriting :exc:`OSError` and :exc:`ValueError` that is raised
247 when an unsupported operation is called on a stream.
248
249
250 .. seealso::
251
252 :mod:`sys`
253 contains the standard IO streams: :data:`sys.stdin`, :data:`sys.stdout`,
254 and :data:`sys.stderr`.
255
256
257 Class hierarchy
258 ---------------
259
260 The implementation of I/O streams is organized as a hierarchy of classes. First
261 :term:`abstract base classes <abstract base class>` (ABCs), which are used to
262 specify the various categories of streams, then concrete classes providing the
263 standard stream implementations.
264
265 .. note::
266
267 The abstract base classes also provide default implementations of some
268 methods in order to help implementation of concrete stream classes. For
269 example, :class:`BufferedIOBase` provides unoptimized implementations of
270 :meth:`!readinto` and :meth:`!readline`.
271
272 At the top of the I/O hierarchy is the abstract base class :class:`IOBase`. It
273 defines the basic interface to a stream. Note, however, that there is no
274 separation between reading and writing to streams; implementations are allowed
275 to raise :exc:`UnsupportedOperation` if they do not support a given operation.
276
277 The :class:`RawIOBase` ABC extends :class:`IOBase`. It deals with the reading
278 and writing of bytes to a stream. :class:`FileIO` subclasses :class:`RawIOBase`
279 to provide an interface to files in the machine's file system.
280
281 The :class:`BufferedIOBase` ABC extends :class:`IOBase`. It deals with
282 buffering on a raw binary stream (:class:`RawIOBase`). Its subclasses,
283 :class:`BufferedWriter`, :class:`BufferedReader`, and :class:`BufferedRWPair`
284 buffer raw binary streams that are writable, readable, and both readable and writable,
285 respectively. :class:`BufferedRandom` provides a buffered interface to seekable streams.
286 Another :class:`BufferedIOBase` subclass, :class:`BytesIO`, is a stream of
287 in-memory bytes.
288
289 The :class:`TextIOBase` ABC extends :class:`IOBase`. It deals with
290 streams whose bytes represent text, and handles encoding and decoding to and
291 from strings. :class:`TextIOWrapper`, which extends :class:`TextIOBase`, is a buffered text
292 interface to a buffered raw stream (:class:`BufferedIOBase`). Finally,
293 :class:`StringIO` is an in-memory stream for text.
294
295 Argument names are not part of the specification, and only the arguments of
296 :func:`open` are intended to be used as keyword arguments.
297
298 The following table summarizes the ABCs provided by the :mod:`io` module:
299
300 .. tabularcolumns:: |l|l|L|L|
301
302 ========================= ================== ======================== ==================================================
303 ABC Inherits Stub Methods Mixin Methods and Properties
304 ========================= ================== ======================== ==================================================
305 :class:`IOBase` ``fileno``, ``seek``, ``close``, ``closed``, ``__enter__``,
306 and ``truncate`` ``__exit__``, ``flush``, ``isatty``, ``__iter__``,
307 ``__next__``, ``readable``, ``readline``,
308 ``readlines``, ``seekable``, ``tell``,
309 ``writable``, and ``writelines``
310 :class:`RawIOBase` :class:`IOBase` ``readinto`` and Inherited :class:`IOBase` methods, ``read``,
311 ``write`` and ``readall``
312 :class:`BufferedIOBase` :class:`IOBase` ``detach``, ``read``, Inherited :class:`IOBase` methods, ``readinto``,
313 ``read1``, and ``write`` and ``readinto1``
314 :class:`TextIOBase` :class:`IOBase` ``detach``, ``read``, Inherited :class:`IOBase` methods, ``encoding``,
315 ``readline``, and ``errors``, and ``newlines``
316 ``write``
317 ========================= ================== ======================== ==================================================
318
319
320 I/O Base Classes
321 ^^^^^^^^^^^^^^^^
322
323 .. class:: IOBase
324
325 The abstract base class for all I/O classes.
326
327 This class provides empty abstract implementations for many methods
328 that derived classes can override selectively; the default
329 implementations represent a file that cannot be read, written or
330 seeked.
331
332 Even though :class:`IOBase` does not declare :meth:`!read`
333 or :meth:`!write` because their signatures will vary, implementations and
334 clients should consider those methods part of the interface. Also,
335 implementations may raise a :exc:`ValueError` (or :exc:`UnsupportedOperation`)
336 when operations they do not support are called.
337
338 The basic type used for binary data read from or written to a file is
339 :class:`bytes`. Other :term:`bytes-like objects <bytes-like object>` are
340 accepted as method arguments too. Text I/O classes work with :class:`str` data.
341
342 Note that calling any method (even inquiries) on a closed stream is
343 undefined. Implementations may raise :exc:`ValueError` in this case.
344
345 :class:`IOBase` (and its subclasses) supports the iterator protocol, meaning
346 that an :class:`IOBase` object can be iterated over yielding the lines in a
347 stream. Lines are defined slightly differently depending on whether the
348 stream is a binary stream (yielding bytes), or a text stream (yielding
349 character strings). See :meth:`~IOBase.readline` below.
350
351 :class:`IOBase` is also a context manager and therefore supports the
352 :keyword:`with` statement. In this example, *file* is closed after the
353 :keyword:`!with` statement's suite is finished---even if an exception occurs::
354
355 with open('spam.txt', 'w') as file:
356 file.write('Spam and eggs!')
357
358 :class:`IOBase` provides these data attributes and methods:
359
360 .. method:: close()
361
362 Flush and close this stream. This method has no effect if the file is
363 already closed. Once the file is closed, any operation on the file
364 (e.g. reading or writing) will raise a :exc:`ValueError`.
365
366 As a convenience, it is allowed to call this method more than once;
367 only the first call, however, will have an effect.
368
369 .. attribute:: closed
370
371 ``True`` if the stream is closed.
372
373 .. method:: fileno()
374
375 Return the underlying file descriptor (an integer) of the stream if it
376 exists. An :exc:`OSError` is raised if the IO object does not use a file
377 descriptor.
378
379 .. method:: flush()
380
381 Flush the write buffers of the stream if applicable. This does nothing
382 for read-only and non-blocking streams.
383
384 .. method:: isatty()
385
386 Return ``True`` if the stream is interactive (i.e., connected to
387 a terminal/tty device).
388
389 .. method:: readable()
390
391 Return ``True`` if the stream can be read from.
392 If ``False``, :meth:`!read` will raise :exc:`OSError`.
393
394 .. method:: readline(size=-1, /)
395
396 Read and return one line from the stream. If *size* is specified, at
397 most *size* bytes will be read.
398
399 The line terminator is always ``b'\n'`` for binary files; for text files,
400 the *newline* argument to :func:`open` can be used to select the line
401 terminator(s) recognized.
402
403 .. method:: readlines(hint=-1, /)
404
405 Read and return a list of lines from the stream. *hint* can be specified
406 to control the number of lines read: no more lines will be read if the
407 total size (in bytes/characters) of all lines so far exceeds *hint*.
408
409 *hint* values of ``0`` or less, as well as ``None``, are treated as no
410 hint.
411
412 Note that it's already possible to iterate on file objects using ``for
413 line in file: ...`` without calling :meth:`!file.readlines`.
414
415 .. method:: seek(offset, whence=os.SEEK_SET, /)
416
417 Change the stream position to the given byte *offset*,
418 interpreted relative to the position indicated by *whence*,
419 and return the new absolute position.
420 Values for *whence* are:
421
422 * :data:`os.SEEK_SET` or ``0`` -- start of the stream (the default);
423 *offset* should be zero or positive
424 * :data:`os.SEEK_CUR` or ``1`` -- current stream position;
425 *offset* may be negative
426 * :data:`os.SEEK_END` or ``2`` -- end of the stream;
427 *offset* is usually negative
428
429 .. versionadded:: 3.1
430 The :data:`!SEEK_*` constants.
431
432 .. versionadded:: 3.3
433 Some operating systems could support additional values, like
434 :const:`os.SEEK_HOLE` or :const:`os.SEEK_DATA`. The valid values
435 for a file could depend on it being open in text or binary mode.
436
437 .. method:: seekable()
438
439 Return ``True`` if the stream supports random access. If ``False``,
440 :meth:`seek`, :meth:`tell` and :meth:`truncate` will raise :exc:`OSError`.
441
442 .. method:: tell()
443
444 Return the current stream position.
445
446 .. method:: truncate(size=None, /)
447
448 Resize the stream to the given *size* in bytes (or the current position
449 if *size* is not specified). The current stream position isn't changed.
450 This resizing can extend or reduce the current file size. In case of
451 extension, the contents of the new file area depend on the platform
452 (on most systems, additional bytes are zero-filled). The new file size
453 is returned.
454
455 .. versionchanged:: 3.5
456 Windows will now zero-fill files when extending.
457
458 .. method:: writable()
459
460 Return ``True`` if the stream supports writing. If ``False``,
461 :meth:`!write` and :meth:`truncate` will raise :exc:`OSError`.
462
463 .. method:: writelines(lines, /)
464
465 Write a list of lines to the stream. Line separators are not added, so it
466 is usual for each of the lines provided to have a line separator at the
467 end.
468
469 .. method:: __del__()
470
471 Prepare for object destruction. :class:`IOBase` provides a default
472 implementation of this method that calls the instance's
473 :meth:`~IOBase.close` method.
474
475
476 .. class:: RawIOBase
477
478 Base class for raw binary streams. It inherits from :class:`IOBase`.
479
480 Raw binary streams typically provide low-level access to an underlying OS
481 device or API, and do not try to encapsulate it in high-level primitives
482 (this functionality is done at a higher-level in buffered binary streams and text streams, described later
483 in this page).
484
485 :class:`RawIOBase` provides these methods in addition to those from
486 :class:`IOBase`:
487
488 .. method:: read(size=-1, /)
489
490 Read up to *size* bytes from the object and return them. As a convenience,
491 if *size* is unspecified or -1, all bytes until EOF are returned.
492
493 Attempts to make only one system call but will retry if interrupted and
494 the signal handler does not raise an exception (see :pep:`475` for the
495 rationale). This means fewer than *size* bytes may be returned if the
496 operating system call returns fewer than *size* bytes.
497
498 If 0 bytes are returned, and *size* was not 0, this indicates end of file.
499 If the object is in non-blocking mode and no bytes are available,
500 ``None`` is returned.
501
502 The default implementation defers to :meth:`readall` and
503 :meth:`readinto`.
504
505 .. method:: readall()
506
507 Read and return all the bytes from the stream until EOF, using multiple
508 calls to the stream if necessary.
509
510 If ``0`` bytes are returned this indicates end of file. If the object is in
511 non-blocking mode and the underlying :meth:`read` returns ``None``
512 indicating no bytes are available, ``None`` is returned.
513
514 .. method:: readinto(b, /)
515
516 Read bytes into a pre-allocated, writable
517 :term:`bytes-like object` *b*, and return the
518 number of bytes read. For example, *b* might be a :class:`bytearray`.
519
520 If ``0`` is returned and ``len(b)`` is not ``0``, this indicates end of file. If
521 the object is in non-blocking mode and no bytes are available, ``None`` is
522 returned.
523
524 .. method:: write(b, /)
525
526 Write the given :term:`bytes-like object`, *b*, to the
527 underlying raw stream, and return the number of
528 bytes written. This can be less than the length of *b* in
529 bytes, depending on specifics of the underlying raw
530 stream, and especially if it is in non-blocking mode. ``None`` is
531 returned if the raw stream is set not to block and no single byte could
532 be readily written to it. The caller may release or mutate *b* after
533 this method returns, so the implementation should only access *b*
534 during the method call.
535
536 .. warning::
537
538 This function does not ensure all bytes are written or an exception is
539 thrown. Callers may implement that behavior by checking the return
540 value and, if it is less than the length of *b*, looping with additional
541 write calls until all unwritten bytes are written. High-level I/O
542 objects like :ref:`binary-io` and :ref:`text-io` implement retry behavior.
543
544 .. class:: BufferedIOBase
545
546 Base class for binary streams that support some kind of buffering.
547 It inherits from :class:`IOBase`.
548
549 The main difference with :class:`RawIOBase` is that methods :meth:`read`,
550 :meth:`readinto` and :meth:`write` will try (respectively) to read
551 as much input as requested or to emit all provided data.
552
553 In addition, if the underlying raw stream is in non-blocking mode, when the
554 system returns would block :meth:`write` will raise :exc:`BlockingIOError`
555 with :attr:`BlockingIOError.characters_written` and :meth:`read` will return
556 data read so far or ``None`` if no data is available.
557
558 Besides, the :meth:`read` method does not have a default
559 implementation that defers to :meth:`readinto`.
560
561 A typical :class:`BufferedIOBase` implementation should not inherit from a
562 :class:`RawIOBase` implementation, but wrap one, like
563 :class:`BufferedWriter` and :class:`BufferedReader` do.
564
565 :class:`BufferedIOBase` provides or overrides these data attributes and
566 methods in addition to those from :class:`IOBase`:
567
568 .. attribute:: raw
569
570 The underlying raw stream (a :class:`RawIOBase` instance) that
571 :class:`BufferedIOBase` deals with. This is not part of the
572 :class:`BufferedIOBase` API and may not exist on some implementations.
573
574 .. method:: detach()
575
576 Separate the underlying raw stream from the buffer and return it.
577
578 After the raw stream has been detached, the buffer is in an unusable
579 state.
580
581 Some buffers, like :class:`BytesIO`, do not have the concept of a single
582 raw stream to return from this method. They raise
583 :exc:`UnsupportedOperation`.
584
585 .. versionadded:: 3.1
586
587 .. method:: read(size=-1, /)
588
589 Read and return up to *size* bytes. If the argument is omitted, ``None``,
590 or negative read as much as possible.
591
592 Fewer bytes may be returned than requested. An empty :class:`bytes` object
593 is returned if the stream is already at EOF. More than one read may be
594 made and calls may be retried if specific errors are encountered, see
595 :meth:`os.read` and :pep:`475` for more details. Less than size bytes
596 being returned does not imply that EOF is imminent.
597
598 When reading as much as possible the default implementation will use
599 ``raw.readall`` if available (which should implement
600 :meth:`RawIOBase.readall`), otherwise will read in a loop until read
601 returns ``None``, an empty :class:`bytes`, or a non-retryable error. For
602 most streams this is to EOF, but for non-blocking streams more data may
603 become available.
604
605 .. note::
606
607 When the underlying raw stream is non-blocking, implementations may
608 either raise :exc:`BlockingIOError` or return ``None`` if no data is
609 available. :mod:`io` implementations return ``None``.
610
611 .. method:: read1(size=-1, /)
612
613 Read and return up to *size* bytes, calling :meth:`~RawIOBase.readinto`
614 which may retry if :py:const:`~errno.EINTR` is encountered per
615 :pep:`475`. If *size* is ``-1`` or not provided, the implementation will
616 choose an arbitrary value for *size*.
617
618 .. note::
619
620 When the underlying raw stream is non-blocking, implementations may
621 either raise :exc:`BlockingIOError` or return ``None`` if no data is
622 available. :mod:`io` implementations return ``None``.
623
624 .. method:: readinto(b, /)
625
626 Read bytes into a pre-allocated, writable
627 :term:`bytes-like object` *b* and return the number of bytes read.
628 For example, *b* might be a :class:`bytearray`.
629
630 Like :meth:`read`, multiple reads may be issued to the underlying raw
631 stream, unless the latter is interactive.
632
633 A :exc:`BlockingIOError` is raised if the underlying raw stream is in non
634 blocking-mode, and has no data available at the moment.
635
636 .. method:: readinto1(b, /)
637
638 Read bytes into a pre-allocated, writable
639 :term:`bytes-like object` *b*, using at most one call to
640 the underlying raw stream's :meth:`~RawIOBase.read` (or
641 :meth:`~RawIOBase.readinto`) method. Return the number of bytes read.
642
643 A :exc:`BlockingIOError` is raised if the underlying raw stream is in non
644 blocking-mode, and has no data available at the moment.
645
646 .. versionadded:: 3.5
647
648 .. method:: write(b, /)
649
650 Write the given :term:`bytes-like object`, *b*, and return the number
651 of bytes written (always equal to the length of *b* in bytes, since if
652 the write fails an :exc:`OSError` will be raised). Depending on the
653 actual implementation, these bytes may be readily written to the
654 underlying stream, or held in a buffer for performance and latency
655 reasons.
656
657 When in non-blocking mode, a :exc:`BlockingIOError` is raised if the
658 data needed to be written to the raw stream but it couldn't accept
659 all the data without blocking.
660
661 The caller may release or mutate *b* after this method returns,
662 so the implementation should only access *b* during the method call.
663
664
665 Raw File I/O
666 ^^^^^^^^^^^^
667
668 .. class:: FileIO(name, mode='r', closefd=True, opener=None)
669
670 A raw binary stream representing an OS-level file containing bytes data. It
671 inherits from :class:`RawIOBase` and implements its low-level access design.
672 This means :meth:`~RawIOBase.write` does not guarantee all bytes are written
673 and :meth:`~RawIOBase.read` may read less bytes than requested even when more
674 bytes may be present in the underlying file. To get "write all" and
675 "read at least" behavior, use :ref:`binary-io`.
676
677 The *name* can be one of two things:
678
679 * a character string or :class:`bytes` object representing the path to the
680 file which will be opened. In this case closefd must be ``True`` (the default)
681 otherwise an error will be raised.
682 * an integer representing the number of an existing OS-level file descriptor
683 to which the resulting :class:`FileIO` object will give access. When the
684 FileIO object is closed this fd will be closed as well, unless *closefd*
685 is set to ``False``.
686
687 The *mode* can be ``'r'``, ``'w'``, ``'x'`` or ``'a'`` for reading
688 (default), writing, exclusive creation or appending. The file will be
689 created if it doesn't exist when opened for writing or appending; it will be
690 truncated when opened for writing. :exc:`FileExistsError` will be raised if
691 it already exists when opened for creating. Opening a file for creating
692 implies writing, so this mode behaves in a similar way to ``'w'``. Add a
693 ``'+'`` to the mode to allow simultaneous reading and writing.
694
695 A custom opener can be used by passing a callable as *opener*. The underlying
696 file descriptor for the file object is then obtained by calling *opener* with
697 (*name*, *flags*). *opener* must return an open file descriptor (passing
698 :mod:`os.open` as *opener* results in functionality similar to passing
699 ``None``).
700
701 The newly created file is :ref:`non-inheritable <fd_inheritance>`.
702
703 See the :func:`open` built-in function for examples on using the *opener*
704 parameter.
705
706 .. warning::
707 :class:`FileIO` is a low-level I/O object and members, such as
708 :meth:`~RawIOBase.read` and :meth:`~RawIOBase.write`, need to have their
709 return values checked explicitly in a retry loop to implement "write all"
710 and "read at least" behavior. High-level I/O objects :ref:`binary-io` and
711 :ref:`text-io` implement retry behavior.
712
713 .. versionchanged:: 3.3
714 The *opener* parameter was added.
715 The ``'x'`` mode was added.
716
717 .. versionchanged:: 3.4
718 The file is now non-inheritable.
719
720 :class:`FileIO` provides these data attributes in addition to those from
721 :class:`RawIOBase` and :class:`IOBase`:
722
723 .. attribute:: mode
724
725 The mode as given in the constructor.
726
727 .. attribute:: name
728
729 The file name. This is the file descriptor of the file when no name is
730 given in the constructor.
731
732
733 Buffered Streams
734 ^^^^^^^^^^^^^^^^
735
736 Buffered I/O streams provide a higher-level interface to an I/O device
737 than raw I/O does.
738
739 .. class:: BytesIO(initial_bytes=b'')
740
741 A binary stream using an in-memory bytes buffer. It inherits from
742 :class:`BufferedIOBase`. The buffer is discarded when the
743 :meth:`~IOBase.close` method is called.
744
745 The optional argument *initial_bytes* is a :term:`bytes-like object` that
746 contains initial data.
747
748 :class:`BytesIO` provides or overrides these methods in addition to those
749 from :class:`BufferedIOBase` and :class:`IOBase`:
750
751 .. method:: getbuffer()
752
753 Return a readable and writable view over the contents of the buffer
754 without copying them. Also, mutating the view will transparently
755 update the contents of the buffer::
756
757 >>> b = io.BytesIO(b"abcdef")
758 >>> view = b.getbuffer()
759 >>> view[2:4] = b"56"
760 >>> b.getvalue()
761 b'ab56ef'
762
763 .. note::
764 As long as the view exists, the :class:`BytesIO` object cannot be
765 resized or closed.
766
767 .. versionadded:: 3.2
768
769 .. method:: getvalue()
770
771 Return :class:`bytes` containing the entire contents of the buffer.
772
773
774 .. method:: read1(size=-1, /)
775
776 In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`.
777
778 .. versionchanged:: 3.7
779 The *size* argument is now optional.
780
781 .. method:: readinto1(b, /)
782
783 In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.readinto`.
784
785 .. versionadded:: 3.5
786
787 .. class:: BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE)
788
789 A buffered binary stream providing higher-level access to a readable, non
790 seekable :class:`RawIOBase` raw binary stream. It inherits from
791 :class:`BufferedIOBase`.
792
793 When reading data from this object, a larger amount of data may be
794 requested from the underlying raw stream, and kept in an internal buffer.
795 The buffered data can then be returned directly on subsequent reads.
796
797 The constructor creates a :class:`BufferedReader` for the given readable
798 *raw* stream and *buffer_size*. If *buffer_size* is omitted,
799 :data:`DEFAULT_BUFFER_SIZE` is used.
800
801 :class:`BufferedReader` provides or overrides these methods in addition to
802 those from :class:`BufferedIOBase` and :class:`IOBase`:
803
804 .. method:: peek(size=0, /)
805
806 Return bytes from the stream without advancing the position. The number of
807 bytes returned may be less or more than requested. If the underlying raw
808 stream is non-blocking and the operation would block, returns empty bytes.
809
810 .. method:: read(size=-1, /)
811
812 In :class:`BufferedReader` this is the same as :meth:`io.BufferedIOBase.read`
813
814 .. method:: read1(size=-1, /)
815
816 In :class:`BufferedReader` this is the same as :meth:`io.BufferedIOBase.read1`
817
818 .. versionchanged:: 3.7
819 The *size* argument is now optional.
820
821
822 .. class:: BufferedWriter(raw, buffer_size=DEFAULT_BUFFER_SIZE)
823
824 A buffered binary stream providing higher-level access to a writeable, non
825 seekable :class:`RawIOBase` raw binary stream. It inherits from
826 :class:`BufferedIOBase`.
827
828 When writing to this object, data is normally placed into an internal
829 buffer. The buffer will be written out to the underlying :class:`RawIOBase`
830 object under various conditions, including:
831
832 * when the buffer gets too small for all pending data;
833 * when :meth:`flush` is called;
834 * when a :meth:`~IOBase.seek` is requested (for :class:`BufferedRandom` objects);
835 * when the :class:`BufferedWriter` object is closed or destroyed.
836
837 The constructor creates a :class:`BufferedWriter` for the given writeable
838 *raw* stream. If the *buffer_size* is not given, it defaults to
839 :data:`DEFAULT_BUFFER_SIZE`.
840
841 :class:`BufferedWriter` provides or overrides these methods in addition to
842 those from :class:`BufferedIOBase` and :class:`IOBase`:
843
844 .. method:: flush()
845
846 Force bytes held in the buffer into the raw stream. A
847 :exc:`BlockingIOError` should be raised if the raw stream blocks.
848
849 .. method:: write(b, /)
850
851 Write the :term:`bytes-like object`, *b*, and return the
852 number of bytes written. When in non-blocking mode, a
853 :exc:`BlockingIOError` with :attr:`BlockingIOError.characters_written` set
854 is raised if the buffer needs to be written out but the raw stream blocks.
855
856
857 .. class:: BufferedRandom(raw, buffer_size=DEFAULT_BUFFER_SIZE)
858
859 A buffered binary stream implementing :class:`BufferedIOBase` interfaces
860 providing higher-level access to a seekable :class:`RawIOBase` raw binary
861 stream.
862
863 The constructor creates a reader and writer for a seekable raw stream, given
864 in the first argument. If the *buffer_size* is omitted it defaults to
865 :data:`DEFAULT_BUFFER_SIZE`.
866
867 :class:`BufferedRandom` is capable of anything :class:`BufferedReader` or
868 :class:`BufferedWriter` can do. In addition, :meth:`~IOBase.seek` and
869 :meth:`~IOBase.tell` are guaranteed to be implemented.
870
871
872 .. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE, /)
873
874 A buffered binary stream providing higher-level access to two non seekable
875 :class:`RawIOBase` raw binary streams---one readable, the other writeable.
876 It inherits from :class:`BufferedIOBase`.
877
878 *reader* and *writer* are :class:`RawIOBase` objects that are readable and
879 writeable respectively. If the *buffer_size* is omitted it defaults to
880 :data:`DEFAULT_BUFFER_SIZE`.
881
882 :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods
883 except for :meth:`~BufferedIOBase.detach`, which raises
884 :exc:`UnsupportedOperation`.
885
886 .. warning::
887
888 :class:`BufferedRWPair` does not attempt to synchronize accesses to
889 its underlying raw streams. You should not pass it the same object
890 as reader and writer; use :class:`BufferedRandom` instead.
891
892
893 Text I/O
894 ^^^^^^^^
895
896 .. class:: TextIOBase
897
898 Base class for text streams. This class provides a character and line based
899 interface to stream I/O. It inherits from :class:`IOBase`.
900
901 :class:`TextIOBase` provides or overrides these data attributes and
902 methods in addition to those from :class:`IOBase`:
903
904 .. attribute:: encoding
905
906 The name of the encoding used to decode the stream's bytes into
907 strings, and to encode strings into bytes.
908
909 .. attribute:: errors
910
911 The error setting of the decoder or encoder.
912
913 .. attribute:: newlines
914
915 A string, a tuple of strings, or ``None``, indicating the newlines
916 translated so far. Depending on the implementation and the initial
917 constructor flags, this may not be available.
918
919 .. attribute:: buffer
920
921 The underlying binary buffer (a :class:`BufferedIOBase`
922 or :class:`RawIOBase` instance) that :class:`TextIOBase` deals with.
923 This is not part of the :class:`TextIOBase` API and may not exist
924 in some implementations.
925
926 .. method:: detach()
927
928 Separate the underlying binary buffer from the :class:`TextIOBase` and
929 return it.
930
931 After the underlying buffer has been detached, the :class:`TextIOBase` is
932 in an unusable state.
933
934 Some :class:`TextIOBase` implementations, like :class:`StringIO`, may not
935 have the concept of an underlying buffer and calling this method will
936 raise :exc:`UnsupportedOperation`.
937
938 .. versionadded:: 3.1
939
940 .. method:: read(size=-1, /)
941
942 Read and return at most *size* characters from the stream as a single
943 :class:`str`. If *size* is negative or ``None``, reads until EOF.
944
945 .. method:: readline(size=-1, /)
946
947 Read until newline or EOF and return a single :class:`str`. If the stream is
948 already at EOF, an empty string is returned.
949
950 If *size* is specified, at most *size* characters will be read.
951
952 .. method:: seek(offset, whence=SEEK_SET, /)
953
954 Change the stream position to the given *offset*. Behaviour depends on
955 the *whence* parameter. The default value for *whence* is
956 :data:`!SEEK_SET`.
957
958 * :data:`!SEEK_SET` or ``0``: seek from the start of the stream
959 (the default); *offset* must either be a number returned by
960 :meth:`TextIOBase.tell`, or zero. Any other *offset* value
961 produces undefined behaviour.
962 * :data:`!SEEK_CUR` or ``1``: "seek" to the current position;
963 *offset* must be zero, which is a no-operation (all other values
964 are unsupported).
965 * :data:`!SEEK_END` or ``2``: seek to the end of the stream;
966 *offset* must be zero (all other values are unsupported).
967
968 Return the new absolute position as an opaque number.
969
970 .. versionadded:: 3.1
971 The :data:`!SEEK_*` constants.
972
973 .. method:: tell()
974
975 Return the current stream position as an opaque number. The number
976 does not usually represent a number of bytes in the underlying
977 binary storage.
978
979 .. method:: write(s, /)
980
981 Write the string *s* to the stream and return the number of characters
982 written.
983
984
985 .. class:: TextIOWrapper(buffer, encoding=None, errors=None, newline=None, \
986 line_buffering=False, write_through=False)
987
988 A buffered text stream providing higher-level access to a
989 :class:`BufferedIOBase` buffered binary stream. It inherits from
990 :class:`TextIOBase`.
991
992 *encoding* gives the name of the encoding that the stream will be decoded or
993 encoded with. In :ref:`UTF-8 Mode <utf8-mode>`, this defaults to UTF-8.
994 Otherwise, it defaults to :func:`locale.getencoding`.
995 ``encoding="locale"`` can be used to specify the current locale's encoding
996 explicitly. See :ref:`io-text-encoding` for more information.
997
998 *errors* is an optional string that specifies how encoding and decoding
999 errors are to be handled. Pass ``'strict'`` to raise a :exc:`ValueError`
1000 exception if there is an encoding error (the default of ``None`` has the same
1001 effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding
1002 errors can lead to data loss.) ``'replace'`` causes a replacement marker
1003 (such as ``'?'``) to be inserted where there is malformed data.
1004 ``'backslashreplace'`` causes malformed data to be replaced by a
1005 backslashed escape sequence. When writing, ``'xmlcharrefreplace'``
1006 (replace with the appropriate XML character reference) or ``'namereplace'``
1007 (replace with ``\N{...}`` escape sequences) can be used. Any other error
1008 handling name that has been registered with
1009 :func:`codecs.register_error` is also valid.
1010
1011 .. index::
1012 single: universal newlines; io.TextIOWrapper class
1013
1014 *newline* controls how line endings are handled. It can be ``None``,
1015 ``''``, ``'\n'``, ``'\r'``, and ``'\r\n'``. It works as follows:
1016
1017 * When reading input from the stream, if *newline* is ``None``,
1018 :term:`universal newlines` mode is enabled. Lines in the input can end in
1019 ``'\n'``, ``'\r'``, or ``'\r\n'``, and these are translated into ``'\n'``
1020 before being returned to the caller. If *newline* is ``''``, universal
1021 newlines mode is enabled, but line endings are returned to the caller
1022 untranslated. If *newline* has any of the other legal values, input lines
1023 are only terminated by the given string, and the line ending is returned to
1024 the caller untranslated.
1025
1026 * When writing output to the stream, if *newline* is ``None``, any ``'\n'``
1027 characters written are translated to the system default line separator,
1028 :data:`os.linesep`. If *newline* is ``''`` or ``'\n'``, no translation
1029 takes place. If *newline* is any of the other legal values, any ``'\n'``
1030 characters written are translated to the given string.
1031
1032 If *line_buffering* is ``True``, :meth:`~IOBase.flush` is implied when a call to
1033 write contains a newline character or a carriage return.
1034
1035 If *write_through* is ``True``, calls to :meth:`~BufferedIOBase.write` are guaranteed
1036 not to be buffered: any data written on the :class:`TextIOWrapper`
1037 object is immediately handled to its underlying binary *buffer*.
1038
1039 .. versionchanged:: 3.3
1040 The *write_through* argument has been added.
1041
1042 .. versionchanged:: 3.3
1043 The default *encoding* is now ``locale.getpreferredencoding(False)``
1044 instead of ``locale.getpreferredencoding()``. Don't change temporary the
1045 locale encoding using :func:`locale.setlocale`, use the current locale
1046 encoding instead of the user preferred encoding.
1047
1048 .. versionchanged:: 3.10
1049 The *encoding* argument now supports the ``"locale"`` dummy encoding name.
1050
1051 :class:`TextIOWrapper` provides these data attributes and methods in
1052 addition to those from :class:`TextIOBase` and :class:`IOBase`:
1053
1054 .. attribute:: line_buffering
1055
1056 Whether line buffering is enabled.
1057
1058 .. attribute:: write_through
1059
1060 Whether writes are passed immediately to the underlying binary
1061 buffer.
1062
1063 .. versionadded:: 3.7
1064
1065 .. method:: reconfigure(*, encoding=None, errors=None, newline=None, \
1066 line_buffering=None, write_through=None)
1067
1068 Reconfigure this text stream using new settings for *encoding*,
1069 *errors*, *newline*, *line_buffering* and *write_through*.
1070
1071 Parameters not specified keep current settings, except
1072 ``errors='strict'`` is used when *encoding* is specified but
1073 *errors* is not specified.
1074
1075 It is not possible to change the encoding or newline if some data
1076 has already been read from the stream. On the other hand, changing
1077 encoding after write is possible.
1078
1079 This method does an implicit stream flush before setting the
1080 new parameters.
1081
1082 .. versionadded:: 3.7
1083
1084 .. versionchanged:: 3.11
1085 The method supports ``encoding="locale"`` option.
1086
1087 .. method:: seek(cookie, whence=os.SEEK_SET, /)
1088
1089 Set the stream position.
1090 Return the new stream position as an :class:`int`.
1091
1092 Four operations are supported,
1093 given by the following argument combinations:
1094
1095 * ``seek(0, SEEK_SET)``: Rewind to the start of the stream.
1096 * ``seek(cookie, SEEK_SET)``: Restore a previous position;
1097 *cookie* **must be** a number returned by :meth:`tell`.
1098 * ``seek(0, SEEK_END)``: Fast-forward to the end of the stream.
1099 * ``seek(0, SEEK_CUR)``: Leave the current stream position unchanged.
1100
1101 Any other argument combinations are invalid,
1102 and may raise exceptions.
1103
1104 .. seealso::
1105
1106 :data:`os.SEEK_SET`, :data:`os.SEEK_CUR`, and :data:`os.SEEK_END`.
1107
1108 .. method:: tell()
1109
1110 Return the stream position as an opaque number.
1111 The return value of :meth:`!tell` can be given as input to :meth:`seek`,
1112 to restore a previous stream position.
1113
1114
1115 .. class:: StringIO(initial_value='', newline='\n')
1116
1117 A text stream using an in-memory text buffer. It inherits from
1118 :class:`TextIOBase`.
1119
1120 The text buffer is discarded when the :meth:`~IOBase.close` method is
1121 called.
1122
1123 The initial value of the buffer can be set by providing *initial_value*.
1124 If newline translation is enabled, newlines will be encoded as if by
1125 :meth:`~TextIOBase.write`. The stream is positioned at the start of the
1126 buffer which emulates opening an existing file in a ``w+`` mode, making it
1127 ready for an immediate write from the beginning or for a write that
1128 would overwrite the initial value. To emulate opening a file in an ``a+``
1129 mode ready for appending, use ``f.seek(0, io.SEEK_END)`` to reposition the
1130 stream at the end of the buffer.
1131
1132 The *newline* argument works like that of :class:`TextIOWrapper`,
1133 except that when writing output to the stream, if *newline* is ``None``,
1134 newlines are written as ``\n`` on all platforms.
1135
1136 :class:`StringIO` provides this method in addition to those from
1137 :class:`TextIOBase` and :class:`IOBase`:
1138
1139 .. method:: getvalue()
1140
1141 Return a :class:`str` containing the entire contents of the buffer.
1142 Newlines are decoded as if by :meth:`~TextIOBase.read`, although
1143 the stream position is not changed.
1144
1145 Example usage::
1146
1147 import io
1148
1149 output = io.StringIO()
1150 output.write('First line.\n')
1151 print('Second line.', file=output)
1152
1153 # Retrieve file contents -- this will be
1154 # 'First line.\nSecond line.\n'
1155 contents = output.getvalue()
1156
1157 # Close object and discard memory buffer --
1158 # .getvalue() will now raise an exception.
1159 output.close()
1160
1161
1162 .. index::
1163 single: universal newlines; io.IncrementalNewlineDecoder class
1164
1165 .. class:: IncrementalNewlineDecoder
1166
1167 A helper codec that decodes newlines for :term:`universal newlines` mode.
1168 It inherits from :class:`codecs.IncrementalDecoder`.
1169
1170
1171 Performance
1172 -----------
1173
1174 This section discusses the performance of the provided concrete I/O
1175 implementations.
1176
1177 Binary I/O
1178 ^^^^^^^^^^
1179
1180 By reading and writing only large chunks of data even when the user asks for a
1181 single byte, buffered I/O hides any inefficiency in calling and executing the
1182 operating system's unbuffered I/O routines. The gain depends on the OS and the
1183 kind of I/O which is performed. For example, on some modern OSes such as Linux,
1184 unbuffered disk I/O can be as fast as buffered I/O. The bottom line, however,
1185 is that buffered I/O offers predictable performance regardless of the platform
1186 and the backing device. Therefore, it is almost always preferable to use
1187 buffered I/O rather than unbuffered I/O for binary data.
1188
1189 Text I/O
1190 ^^^^^^^^
1191
1192 Text I/O over a binary storage (such as a file) is significantly slower than
1193 binary I/O over the same storage, because it requires conversions between
1194 unicode and binary data using a character codec. This can become noticeable
1195 handling huge amounts of text data like large log files. Also,
1196 :meth:`~TextIOBase.tell` and :meth:`~TextIOBase.seek` are both quite slow
1197 due to the reconstruction algorithm used.
1198
1199 :class:`StringIO`, however, is a native in-memory unicode container and will
1200 exhibit similar speed to :class:`BytesIO`.
1201
1202 Multi-threading
1203 ^^^^^^^^^^^^^^^
1204
1205 :class:`FileIO` objects are thread-safe to the extent that the operating system
1206 calls (such as :manpage:`read(2)` under Unix) they wrap are thread-safe too.
1207
1208 Binary buffered objects (instances of :class:`BufferedReader`,
1209 :class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`)
1210 protect their internal structures using a lock; it is therefore safe to call
1211 them from multiple threads at once.
1212
1213 :class:`TextIOWrapper` objects are not thread-safe.
1214
1215 Reentrancy
1216 ^^^^^^^^^^
1217
1218 Binary buffered objects (instances of :class:`BufferedReader`,
1219 :class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`)
1220 are not reentrant. While reentrant calls will not happen in normal situations,
1221 they can arise from doing I/O in a :mod:`signal` handler. If a thread tries to
1222 re-enter a buffered object which it is already accessing, a :exc:`RuntimeError`
1223 is raised. Note this doesn't prohibit a different thread from entering the
1224 buffered object.
1225
1226 The above implicitly extends to text files, since the :func:`open` function
1227 will wrap a buffered object inside a :class:`TextIOWrapper`. This includes
1228 standard streams and therefore affects the built-in :func:`print` function as
1229 well.