| 1 |
.. _guide_migration_s3: |
| 2 |
|
| 3 |
Amazon S3 |
| 4 |
========= |
| 5 |
Boto 2.x contains a number of customizations to make working with Amazon S3 buckets and keys easy. Boto3 exposes these same objects through its resources interface in a unified and consistent way. |
| 6 |
|
| 7 |
Creating the connection |
| 8 |
----------------------- |
| 9 |
Boto3 has both low-level clients and higher-level resources. For Amazon S3, the higher-level resources are the most similar to Boto 2.x's ``s3`` module:: |
| 10 |
|
| 11 |
# Boto 2.x |
| 12 |
import boto |
| 13 |
s3_connection = boto.connect_s3() |
| 14 |
|
| 15 |
# Boto3 |
| 16 |
import boto3 |
| 17 |
s3 = boto3.resource('s3') |
| 18 |
|
| 19 |
Creating a bucket |
| 20 |
----------------- |
| 21 |
Creating a bucket in Boto 2 and Boto3 is very similar, except that in Boto3 all action parameters must be passed via keyword arguments and a bucket configuration must be specified manually:: |
| 22 |
|
| 23 |
# Boto 2.x |
| 24 |
s3_connection.create_bucket('amzn-s3-demo-bucket') |
| 25 |
s3_connection.create_bucket('amzn-s3-demo-bucket', location=Location.USWest) |
| 26 |
|
| 27 |
# Boto3 |
| 28 |
s3.create_bucket(Bucket='amzn-s3-demo-bucket') |
| 29 |
s3.create_bucket(Bucket='amzn-s3-demo-bucket', CreateBucketConfiguration={ |
| 30 |
'LocationConstraint': 'us-west-1'}) |
| 31 |
|
| 32 |
Storing data |
| 33 |
------------ |
| 34 |
Storing data from a file, stream, or string is easy:: |
| 35 |
|
| 36 |
# Boto 2.x |
| 37 |
from boto.s3.key import Key |
| 38 |
key = Key('hello.txt') |
| 39 |
key.set_contents_from_file('/tmp/hello.txt') |
| 40 |
|
| 41 |
# Boto3 |
| 42 |
s3.Object('amzn-s3-demo-bucket', 'hello.txt').put(Body=open('/tmp/hello.txt', 'rb')) |
| 43 |
|
| 44 |
|
| 45 |
Accessing a bucket |
| 46 |
------------------ |
| 47 |
Getting a bucket is easy with Boto3's resources, however these do not automatically validate whether a bucket exists:: |
| 48 |
|
| 49 |
# Boto 2.x |
| 50 |
bucket = s3_connection.get_bucket('amzn-s3-demo-bucket', validate=False) |
| 51 |
exists = s3_connection.lookup('amzn-s3-demo-bucket') |
| 52 |
|
| 53 |
# Boto3 |
| 54 |
import botocore |
| 55 |
bucket = s3.Bucket('amzn-s3-demo-bucket') |
| 56 |
exists = True |
| 57 |
try: |
| 58 |
s3.meta.client.head_bucket(Bucket='amzn-s3-demo-bucket') |
| 59 |
except botocore.exceptions.ClientError as e: |
| 60 |
# If a client error is thrown, then check that it was a 404 error. |
| 61 |
# If it was a 404 error, then the bucket does not exist. |
| 62 |
error_code = e.response['Error']['Code'] |
| 63 |
if error_code == '404': |
| 64 |
exists = False |
| 65 |
|
| 66 |
Deleting a bucket |
| 67 |
----------------- |
| 68 |
All of the keys in a bucket must be deleted before the bucket itself can be deleted:: |
| 69 |
|
| 70 |
# Boto 2.x |
| 71 |
for key in bucket: |
| 72 |
key.delete() |
| 73 |
bucket.delete() |
| 74 |
|
| 75 |
# Boto3 |
| 76 |
for key in bucket.objects.all(): |
| 77 |
key.delete() |
| 78 |
bucket.delete() |
| 79 |
|
| 80 |
Iteration of buckets and keys |
| 81 |
----------------------------- |
| 82 |
Bucket and key objects are no longer iterable, but now provide collection attributes which can be iterated:: |
| 83 |
|
| 84 |
# Boto 2.x |
| 85 |
for bucket in s3_connection: |
| 86 |
for key in bucket: |
| 87 |
print(key.name) |
| 88 |
|
| 89 |
# Boto3 |
| 90 |
for bucket in s3.buckets.all(): |
| 91 |
for key in bucket.objects.all(): |
| 92 |
print(key.key) |
| 93 |
|
| 94 |
Access controls |
| 95 |
--------------- |
| 96 |
Getting and setting canned access control values in Boto3 operates on an ``ACL`` resource object:: |
| 97 |
|
| 98 |
# Boto 2.x |
| 99 |
bucket.set_acl('public-read') |
| 100 |
key.set_acl('public-read') |
| 101 |
|
| 102 |
# Boto3 |
| 103 |
bucket.Acl().put(ACL='public-read') |
| 104 |
obj.Acl().put(ACL='public-read') |
| 105 |
|
| 106 |
It's also possible to retrieve the policy grant information:: |
| 107 |
|
| 108 |
# Boto 2.x |
| 109 |
acp = bucket.get_acl() |
| 110 |
for grant in acp.acl.grants: |
| 111 |
print(grant.display_name, grant.permission) |
| 112 |
|
| 113 |
# Boto3 |
| 114 |
acl = bucket.Acl() |
| 115 |
for grant in acl.grants: |
| 116 |
print(grant['Grantee']['DisplayName'], grant['Permission']) |
| 117 |
|
| 118 |
Boto3 lacks the grant shortcut methods present in Boto 2.x, but it is still fairly simple to add grantees:: |
| 119 |
|
| 120 |
# Boto 2.x |
| 121 |
bucket.add_email_grant('READ', '[email protected]') |
| 122 |
|
| 123 |
# Boto3 |
| 124 |
bucket.Acl.put(GrantRead='[email protected]') |
| 125 |
|
| 126 |
Key metadata |
| 127 |
------------ |
| 128 |
It's possible to set arbitrary metadata on keys:: |
| 129 |
|
| 130 |
# Boto 2.x |
| 131 |
key.set_metadata('meta1', 'This is my metadata value') |
| 132 |
print(key.get_metadata('meta1')) |
| 133 |
|
| 134 |
# Boto3 |
| 135 |
key.put(Metadata={'meta1': 'This is my metadata value'}) |
| 136 |
print(key.metadata['meta1']) |
| 137 |
|
| 138 |
Managing CORS configurations |
| 139 |
--------------------------- |
| 140 |
Allows you to manage the cross-origin resource sharing configuration for S3 buckets:: |
| 141 |
|
| 142 |
# Boto 2.x |
| 143 |
cors = bucket.get_cors() |
| 144 |
|
| 145 |
config = CORSConfiguration() |
| 146 |
config.add_rule('GET', '*') |
| 147 |
bucket.set_cors(config) |
| 148 |
|
| 149 |
bucket.delete_cors() |
| 150 |
|
| 151 |
# Boto3 |
| 152 |
cors = bucket.Cors() |
| 153 |
|
| 154 |
config = { |
| 155 |
'CORSRules': [ |
| 156 |
{ |
| 157 |
'AllowedMethods': ['GET'], |
| 158 |
'AllowedOrigins': ['*'] |
| 159 |
} |
| 160 |
] |
| 161 |
} |
| 162 |
cors.put(CORSConfiguration=config) |
| 163 |
|
| 164 |
cors.delete() |