| 1 |
.. Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. |
| 2 |
|
| 3 |
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 |
| 4 |
International License (the "License"). You may not use this file except in compliance with the |
| 5 |
License. A copy of the License is located at http://creativecommons.org/licenses/by-nc-sa/4.0/. |
| 6 |
|
| 7 |
This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, |
| 8 |
either express or implied. See the License for the specific language governing permissions and |
| 9 |
limitations under the License. |
| 10 |
|
| 11 |
|
| 12 |
########################### |
| 13 |
File transfer configuration |
| 14 |
########################### |
| 15 |
|
| 16 |
When uploading, downloading, or copying a file or S3 object, the AWS SDK for |
| 17 |
Python automatically manages retries and multipart and non-multipart transfers. |
| 18 |
|
| 19 |
The management operations are performed by using reasonable default settings |
| 20 |
that are well-suited for most scenarios. To handle a special case, the default |
| 21 |
settings can be configured to meet requirements. |
| 22 |
|
| 23 |
Configuration settings are stored in a |
| 24 |
:py:class:`boto3.s3.transfer.TransferConfig` object. The object is passed to |
| 25 |
a transfer method (``upload_file``, ``download_file``, etc.) in the ``Config=`` |
| 26 |
parameter. |
| 27 |
|
| 28 |
The remaining sections demonstrate how to configure various transfer operations |
| 29 |
with the ``TransferConfig`` object. |
| 30 |
|
| 31 |
|
| 32 |
Multipart transfers |
| 33 |
=================== |
| 34 |
|
| 35 |
Multipart transfers occur when the file size exceeds the value of the |
| 36 |
``multipart_threshold`` attribute. |
| 37 |
|
| 38 |
The following example configures an ``upload_file`` transfer to be multipart |
| 39 |
if the file size is larger than the threshold specified in the |
| 40 |
``TransferConfig`` object. |
| 41 |
|
| 42 |
.. code-block:: python |
| 43 |
|
| 44 |
import boto3 |
| 45 |
from boto3.s3.transfer import TransferConfig |
| 46 |
|
| 47 |
# Set the desired multipart threshold value (5GB) |
| 48 |
GB = 1024 ** 3 |
| 49 |
config = TransferConfig(multipart_threshold=5*GB) |
| 50 |
|
| 51 |
# Perform the transfer |
| 52 |
s3 = boto3.client('s3') |
| 53 |
s3.upload_file('FILE_NAME', 'amzn-s3-demo-bucket', 'OBJECT_NAME', Config=config) |
| 54 |
|
| 55 |
|
| 56 |
Concurrent transfer operations |
| 57 |
============================== |
| 58 |
|
| 59 |
The maximum number of concurrent S3 API transfer operations can be tuned to |
| 60 |
adjust for the connection speed. Set the ``max_concurrency`` attribute to |
| 61 |
increase or decrease bandwidth usage. |
| 62 |
|
| 63 |
The attribute's default setting is 10. To reduce bandwidth usage, reduce the |
| 64 |
value; to increase usage, increase it. |
| 65 |
|
| 66 |
.. code-block:: python |
| 67 |
|
| 68 |
# To consume less downstream bandwidth, decrease the maximum concurrency |
| 69 |
config = TransferConfig(max_concurrency=5) |
| 70 |
|
| 71 |
# Download an S3 object |
| 72 |
s3 = boto3.client('s3') |
| 73 |
s3.download_file('amzn-s3-demo-bucket', 'OBJECT_NAME', 'FILE_NAME', Config=config) |
| 74 |
|
| 75 |
|
| 76 |
Threads |
| 77 |
======= |
| 78 |
|
| 79 |
Transfer operations use threads to implement concurrency. Thread use can be |
| 80 |
disabled by setting the ``use_threads`` attribute to ``False.`` |
| 81 |
|
| 82 |
If thread use is disabled, transfer concurrency does not occur. Accordingly, |
| 83 |
the value of the ``max_concurrency`` attribute is ignored. |
| 84 |
|
| 85 |
.. code-block:: python |
| 86 |
|
| 87 |
# Disable thread use/transfer concurrency |
| 88 |
config = TransferConfig(use_threads=False) |
| 89 |
|
| 90 |
s3 = boto3.client('s3') |
| 91 |
s3.download_file('amzn-s3-demo-bucket', 'OBJECT_NAME', 'FILE_NAME', Config=config) |