Debian Codegraph @

docs/source/guide/s3-uploading-files.rst

raw
1 .. Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2
3 This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0
4 International License (the "License"). You may not use this file except in compliance with the
5 License. A copy of the License is located at http://creativecommons.org/licenses/by-nc-sa/4.0/.
6
7 This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
8 either express or implied. See the License for the specific language governing permissions and
9 limitations under the License.
10
11
12 ###############
13 Uploading files
14 ###############
15
16 The AWS SDK for Python provides a pair of methods to upload a file to an S3
17 bucket.
18
19 The ``upload_file`` method accepts a file name, a bucket name, and an object
20 name. The method handles large files by splitting them into smaller chunks
21 and uploading each chunk in parallel.
22
23 .. code-block:: python
24
25 import logging
26 import boto3
27 from botocore.exceptions import ClientError
28 import os
29
30
31 def upload_file(file_name, bucket, object_name=None):
32 """Upload a file to an S3 bucket
33
34 :param file_name: File to upload
35 :param bucket: Bucket to upload to
36 :param object_name: S3 object name. If not specified then file_name is used
37 :return: True if file was uploaded, else False
38 """
39
40 # If S3 object_name was not specified, use file_name
41 if object_name is None:
42 object_name = os.path.basename(file_name)
43
44 # Upload the file
45 s3_client = boto3.client('s3')
46 try:
47 response = s3_client.upload_file(file_name, bucket, object_name)
48 except ClientError as e:
49 logging.error(e)
50 return False
51 return True
52
53
54 The ``upload_fileobj`` method accepts a readable file-like object. The file
55 object must be opened in binary mode, not text mode.
56
57 .. code-block:: python
58
59 s3 = boto3.client('s3')
60 with open("FILE_NAME", "rb") as f:
61 s3.upload_fileobj(f, "amzn-s3-demo-bucket", "OBJECT_NAME")
62
63
64 The ``upload_file`` and ``upload_fileobj`` methods are provided by the S3
65 ``Client``, ``Bucket``, and ``Object`` classes. The method functionality
66 provided by each class is identical. No benefits are gained by calling one
67 class's method over another's. Use whichever class is most convenient.
68
69
70 The ExtraArgs parameter
71 ===========================
72
73 Both ``upload_file`` and ``upload_fileobj`` accept an optional ``ExtraArgs``
74 parameter that can be used for various purposes. The list of valid
75 ``ExtraArgs`` settings is specified in the ``ALLOWED_UPLOAD_ARGS`` attribute
76 of the ``S3Transfer`` object
77 at :py:attr:`boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS`.
78
79 The following ``ExtraArgs`` setting specifies metadata to attach to the S3
80 object.
81
82 .. code-block:: python
83
84 s3.upload_file(
85 'FILE_NAME', 'amzn-s3-demo-bucket', 'OBJECT_NAME',
86 ExtraArgs={'Metadata': {'mykey': 'myvalue'}}
87 )
88
89
90 The following ``ExtraArgs`` setting assigns the canned ACL (access control
91 list) value 'public-read' to the S3 object.
92
93 .. code-block:: python
94
95 s3.upload_file(
96 'FILE_NAME', 'amzn-s3-demo-bucket', 'OBJECT_NAME',
97 ExtraArgs={'ACL': 'public-read'}
98 )
99
100
101 The ``ExtraArgs`` parameter can also be used to set custom or multiple ACLs.
102
103 .. code-block:: python
104
105 s3.upload_file(
106 'FILE_NAME', 'amzn-s3-demo-bucket', 'OBJECT_NAME',
107 ExtraArgs={
108 'GrantRead': 'uri="http://acs.amazonaws.com/groups/global/AllUsers"',
109 'GrantFullControl': 'id="01234567890abcdefg"',
110 }
111 )
112
113
114 The Callback parameter
115 ==========================
116
117 Both ``upload_file`` and ``upload_fileobj`` accept an optional ``Callback``
118 parameter. The parameter references a class that the Python SDK invokes
119 intermittently during the transfer operation.
120
121 Invoking a Python class executes the class's ``__call__`` method. For each
122 invocation, the class is passed the number of bytes transferred up
123 to that point. This information can be used to implement a progress monitor.
124
125 The following ``Callback`` setting instructs the Python SDK to create an
126 instance of the ``ProgressPercentage`` class. During the upload, the
127 instance's ``__call__`` method will be invoked intermittently.
128
129 .. code-block:: python
130
131 s3.upload_file(
132 'FILE_NAME', 'amzn-s3-demo-bucket', 'OBJECT_NAME',
133 Callback=ProgressPercentage('FILE_NAME')
134 )
135
136
137 An example implementation of the ``ProcessPercentage`` class is shown below.
138
139 .. code-block:: python
140
141 import os
142 import sys
143 import threading
144
145 class ProgressPercentage(object):
146
147 def __init__(self, filename):
148 self._filename = filename
149 self._size = float(os.path.getsize(filename))
150 self._seen_so_far = 0
151 self._lock = threading.Lock()
152
153 def __call__(self, bytes_amount):
154 # To simplify, assume this is hooked up to a single filename
155 with self._lock:
156 self._seen_so_far += bytes_amount
157 percentage = (self._seen_so_far / self._size) * 100
158 sys.stdout.write(
159 "\r%s %s / %s (%.2f%%)" % (
160 self._filename, self._seen_so_far, self._size,
161 percentage))
162 sys.stdout.flush()