Debian Codegraph @

docs/source/guide/s3.rst

raw
1 .. Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2
3 This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0
4 International License (the "License"). You may not use this file except in compliance with the
5 License. A copy of the License is located at http://creativecommons.org/licenses/by-nc-sa/4.0/.
6
7 This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
8 either express or implied. See the License for the specific language governing permissions and
9 limitations under the License.
10
11
12 ###########################
13 File transfer configuration
14 ###########################
15
16 When uploading, downloading, or copying a file or S3 object, the AWS SDK for
17 Python automatically manages retries and multipart and non-multipart transfers.
18
19 The management operations are performed by using reasonable default settings
20 that are well-suited for most scenarios. To handle a special case, the default
21 settings can be configured to meet requirements.
22
23 Configuration settings are stored in a
24 :py:class:`boto3.s3.transfer.TransferConfig` object. The object is passed to
25 a transfer method (``upload_file``, ``download_file``, etc.) in the ``Config=``
26 parameter.
27
28 The remaining sections demonstrate how to configure various transfer operations
29 with the ``TransferConfig`` object.
30
31
32 Multipart transfers
33 ===================
34
35 Multipart transfers occur when the file size exceeds the value of the
36 ``multipart_threshold`` attribute.
37
38 The following example configures an ``upload_file`` transfer to be multipart
39 if the file size is larger than the threshold specified in the
40 ``TransferConfig`` object.
41
42 .. code-block:: python
43
44 import boto3
45 from boto3.s3.transfer import TransferConfig
46
47 # Set the desired multipart threshold value (5GB)
48 GB = 1024 ** 3
49 config = TransferConfig(multipart_threshold=5*GB)
50
51 # Perform the transfer
52 s3 = boto3.client('s3')
53 s3.upload_file('FILE_NAME', 'amzn-s3-demo-bucket', 'OBJECT_NAME', Config=config)
54
55
56 Concurrent transfer operations
57 ==============================
58
59 The maximum number of concurrent S3 API transfer operations can be tuned to
60 adjust for the connection speed. Set the ``max_concurrency`` attribute to
61 increase or decrease bandwidth usage.
62
63 The attribute's default setting is 10. To reduce bandwidth usage, reduce the
64 value; to increase usage, increase it.
65
66 .. code-block:: python
67
68 # To consume less downstream bandwidth, decrease the maximum concurrency
69 config = TransferConfig(max_concurrency=5)
70
71 # Download an S3 object
72 s3 = boto3.client('s3')
73 s3.download_file('amzn-s3-demo-bucket', 'OBJECT_NAME', 'FILE_NAME', Config=config)
74
75
76 Threads
77 =======
78
79 Transfer operations use threads to implement concurrency. Thread use can be
80 disabled by setting the ``use_threads`` attribute to ``False.``
81
82 If thread use is disabled, transfer concurrency does not occur. Accordingly,
83 the value of the ``max_concurrency`` attribute is ignored.
84
85 .. code-block:: python
86
87 # Disable thread use/transfer concurrency
88 config = TransferConfig(use_threads=False)
89
90 s3 = boto3.client('s3')
91 s3.download_file('amzn-s3-demo-bucket', 'OBJECT_NAME', 'FILE_NAME', Config=config)