| 1 |
List objects in an Amazon S3 bucket |
| 2 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 3 |
|
| 4 |
The following example shows how to use an Amazon S3 bucket resource to list |
| 5 |
the objects in the bucket. |
| 6 |
|
| 7 |
.. code-block:: python |
| 8 |
|
| 9 |
import boto3 |
| 10 |
|
| 11 |
s3 = boto3.resource('s3') |
| 12 |
bucket = s3.Bucket('amzn-s3-demo-bucket') |
| 13 |
for obj in bucket.objects.all(): |
| 14 |
print(obj.key) |
| 15 |
|
| 16 |
|
| 17 |
List top-level common prefixes in Amazon S3 bucket |
| 18 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 19 |
|
| 20 |
This example shows how to list all of the top-level common prefixes in an |
| 21 |
Amazon S3 bucket: |
| 22 |
|
| 23 |
.. code-block:: python |
| 24 |
|
| 25 |
import boto3 |
| 26 |
|
| 27 |
client = boto3.client('s3') |
| 28 |
paginator = client.get_paginator('list_objects') |
| 29 |
result = paginator.paginate(Bucket='amzn-s3-demo-bucket', Delimiter='/') |
| 30 |
for prefix in result.search('CommonPrefixes'): |
| 31 |
print(prefix.get('Prefix')) |
| 32 |
|
| 33 |
|
| 34 |
Restore Glacier objects in an Amazon S3 bucket |
| 35 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 36 |
|
| 37 |
The following example shows how to initiate restoration of glacier objects in |
| 38 |
an Amazon S3 bucket, determine if a restoration is on-going, and determine if a |
| 39 |
restoration is finished. |
| 40 |
|
| 41 |
.. code-block:: python |
| 42 |
|
| 43 |
import boto3 |
| 44 |
|
| 45 |
s3 = boto3.resource('s3') |
| 46 |
bucket = s3.Bucket('amzn-s3-demo-bucket') |
| 47 |
for obj_sum in bucket.objects.all(): |
| 48 |
obj = s3.Object(obj_sum.bucket_name, obj_sum.key) |
| 49 |
if obj.storage_class == 'GLACIER': |
| 50 |
# Try to restore the object if the storage class is glacier and |
| 51 |
# the object does not have a completed or ongoing restoration |
| 52 |
# request. |
| 53 |
if obj.restore is None: |
| 54 |
print('Submitting restoration request: %s' % obj.key) |
| 55 |
obj.restore_object(RestoreRequest={'Days': 1}) |
| 56 |
# Print out objects whose restoration is on-going |
| 57 |
elif 'ongoing-request="true"' in obj.restore: |
| 58 |
print('Restoration in-progress: %s' % obj.key) |
| 59 |
# Print out objects whose restoration is complete |
| 60 |
elif 'ongoing-request="false"' in obj.restore: |
| 61 |
print('Restoration complete: %s' % obj.key) |
| 62 |
|
| 63 |
|
| 64 |
Uploading/downloading files using SSE KMS |
| 65 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 66 |
|
| 67 |
This example shows how to use SSE-KMS to upload objects using |
| 68 |
server side encryption with a key managed by KMS. |
| 69 |
|
| 70 |
We can either use the default KMS master key, or create a |
| 71 |
custom key in AWS and use it to encrypt the object by passing in its |
| 72 |
key id. |
| 73 |
|
| 74 |
With KMS, nothing else needs to be provided for getting the |
| 75 |
object; S3 already knows how to decrypt the object. |
| 76 |
|
| 77 |
|
| 78 |
.. code-block:: python |
| 79 |
|
| 80 |
import boto3 |
| 81 |
import os |
| 82 |
|
| 83 |
BUCKET = 'amzn-s3-demo-bucket' |
| 84 |
s3 = boto3.client('s3') |
| 85 |
keyid = '<the key id>' |
| 86 |
|
| 87 |
print("Uploading S3 object with SSE-KMS") |
| 88 |
s3.put_object(Bucket=BUCKET, |
| 89 |
Key='encrypt-key', |
| 90 |
Body=b'foobar', |
| 91 |
ServerSideEncryption='aws:kms', |
| 92 |
# Optional: SSEKMSKeyId |
| 93 |
SSEKMSKeyId=keyid) |
| 94 |
print("Done") |
| 95 |
|
| 96 |
# Getting the object: |
| 97 |
print("Getting S3 object...") |
| 98 |
response = s3.get_object(Bucket=BUCKET, |
| 99 |
Key='encrypt-key') |
| 100 |
print("Done, response body:") |
| 101 |
print(response['Body'].read()) |
| 102 |
|
| 103 |
|
| 104 |
Uploading/downloading files using SSE Customer Keys |
| 105 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 106 |
|
| 107 |
This example shows how to use SSE-C to upload objects using |
| 108 |
server side encryption with a customer provided key. |
| 109 |
|
| 110 |
First, we'll need a 32 byte key. For this example, we'll |
| 111 |
randomly generate a key but you can use any 32 byte key |
| 112 |
you want. Remember, you must the same key to download |
| 113 |
the object. If you lose the encryption key, you lose |
| 114 |
the object. |
| 115 |
|
| 116 |
Also note how we don't have to provide the SSECustomerKeyMD5. |
| 117 |
Boto3 will automatically compute this value for us. |
| 118 |
|
| 119 |
|
| 120 |
.. code-block:: python |
| 121 |
|
| 122 |
import boto3 |
| 123 |
import os |
| 124 |
|
| 125 |
BUCKET = 'amzn-s3-demo-bucket' |
| 126 |
KEY = os.urandom(32) |
| 127 |
s3 = boto3.client('s3') |
| 128 |
|
| 129 |
print("Uploading S3 object with SSE-C") |
| 130 |
s3.put_object(Bucket=BUCKET, |
| 131 |
Key='encrypt-key', |
| 132 |
Body=b'foobar', |
| 133 |
SSECustomerKey=KEY, |
| 134 |
SSECustomerAlgorithm='AES256') |
| 135 |
print("Done") |
| 136 |
|
| 137 |
# Getting the object: |
| 138 |
print("Getting S3 object...") |
| 139 |
# Note how we're using the same ``KEY`` we |
| 140 |
# created earlier. |
| 141 |
response = s3.get_object(Bucket=BUCKET, |
| 142 |
Key='encrypt-key', |
| 143 |
SSECustomerKey=KEY, |
| 144 |
SSECustomerAlgorithm='AES256') |
| 145 |
print("Done, response body:") |
| 146 |
print(response['Body'].read()) |
| 147 |
|
| 148 |
|
| 149 |
Downloading a specific version of an S3 object |
| 150 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 151 |
|
| 152 |
This example shows how to download a specific version of an |
| 153 |
S3 object. |
| 154 |
|
| 155 |
.. code-block:: python |
| 156 |
|
| 157 |
import boto3 |
| 158 |
s3 = boto3.client('s3') |
| 159 |
|
| 160 |
s3.download_file( |
| 161 |
"amzn-s3-demo-bucket", "key-name", "tmp.txt", |
| 162 |
ExtraArgs={"VersionId": "my-version-id"} |
| 163 |
) |
| 164 |
|
| 165 |
|
| 166 |
Filter objects by last modified time using JMESPath |
| 167 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 168 |
|
| 169 |
This example shows how to filter objects by last modified time |
| 170 |
using JMESPath. |
| 171 |
|
| 172 |
.. code-block:: python |
| 173 |
|
| 174 |
import boto3 |
| 175 |
s3 = boto3.client("s3") |
| 176 |
|
| 177 |
s3_paginator = s3.get_paginator('list_objects_v2') |
| 178 |
s3_iterator = s3_paginator.paginate(Bucket='amzn-s3-demo-bucket') |
| 179 |
|
| 180 |
filtered_iterator = s3_iterator.search( |
| 181 |
"Contents[?to_string(LastModified)>='\"2022-01-05 08:05:37+00:00\"'].Key" |
| 182 |
) |
| 183 |
|
| 184 |
for key_data in filtered_iterator: |
| 185 |
print(key_data) |