| 1 |
.. _guide_collections: |
| 2 |
|
| 3 |
Collections |
| 4 |
=========== |
| 5 |
|
| 6 |
Overview |
| 7 |
-------- |
| 8 |
A collection provides an iterable interface to a group of resources. |
| 9 |
Collections behave similarly to |
| 10 |
`Django QuerySets <https://docs.djangoproject.com/en/4.2/ref/models/querysets/>`_ |
| 11 |
and expose a similar API. A collection seamlessly handles pagination for |
| 12 |
you, making it possible to easily iterate over all items from all pages of |
| 13 |
data. Example of a collection:: |
| 14 |
|
| 15 |
# SQS list all queues |
| 16 |
sqs = boto3.resource('sqs') |
| 17 |
for queue in sqs.queues.all(): |
| 18 |
print(queue.url) |
| 19 |
|
| 20 |
When collections make requests |
| 21 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 22 |
Collections can be created and manipulated without any request being made |
| 23 |
to the underlying service. A collection makes a remote service request under |
| 24 |
the following conditions: |
| 25 |
|
| 26 |
* **Iteration**:: |
| 27 |
|
| 28 |
for bucket in s3.buckets.all(): |
| 29 |
print(bucket.name) |
| 30 |
|
| 31 |
* **Conversion to list()**:: |
| 32 |
|
| 33 |
buckets = list(s3.buckets.all()) |
| 34 |
|
| 35 |
* **Batch actions (see below)**:: |
| 36 |
|
| 37 |
s3.Bucket('amzn-s3-demo-bucket').objects.delete() |
| 38 |
|
| 39 |
Filtering |
| 40 |
--------- |
| 41 |
Some collections support extra arguments to filter the returned data set, |
| 42 |
which are passed into the underlying service operation. Use the |
| 43 |
:py:meth:`~boto3.resources.collection.Collection.filter` method to filter |
| 44 |
the results:: |
| 45 |
|
| 46 |
# S3 list all keys with the prefix 'photos/' |
| 47 |
s3 = boto3.resource('s3') |
| 48 |
for bucket in s3.buckets.all(): |
| 49 |
for obj in bucket.objects.filter(Prefix='photos/'): |
| 50 |
print('{0}:{1}'.format(bucket.name, obj.key)) |
| 51 |
|
| 52 |
.. warning:: |
| 53 |
|
| 54 |
Behind the scenes, the above example will call ``ListBuckets``, |
| 55 |
``ListObjects``, and ``HeadObject`` many times. If you have a large |
| 56 |
number of S3 objects then this could incur a significant cost. |
| 57 |
|
| 58 |
Chainability |
| 59 |
------------ |
| 60 |
Collection methods are chainable. They return copies of the collection |
| 61 |
rather than modifying the collection, including a deep copy of any |
| 62 |
associated operation parameters. For example, this allows you |
| 63 |
to build up multiple collections from a base which they all have |
| 64 |
in common:: |
| 65 |
|
| 66 |
# EC2 find instances |
| 67 |
ec2 = boto3.resource('ec2') |
| 68 |
base = ec2.instances.filter(InstanceIds=['id1', 'id2', 'id3']) |
| 69 |
|
| 70 |
filters = [{ |
| 71 |
'Name': 'tenancy', |
| 72 |
'Values': ['dedicated'] |
| 73 |
}] |
| 74 |
filtered1 = base.filter(Filters=filters) |
| 75 |
|
| 76 |
# Note, this does NOT modify the filters in ``filtered1``! |
| 77 |
filters.append({'name': 'instance-type', 'value': 't1.micro'}) |
| 78 |
filtered2 = base.filter(Filters=filters) |
| 79 |
|
| 80 |
print('All instances:') |
| 81 |
for instance in base: |
| 82 |
print(instance.id) |
| 83 |
|
| 84 |
print('Dedicated instances:') |
| 85 |
for instance in filtered1: |
| 86 |
print(instance.id) |
| 87 |
|
| 88 |
print('Dedicated micro instances:') |
| 89 |
for instance in filtered2: |
| 90 |
print(instance.id) |
| 91 |
|
| 92 |
Limiting results |
| 93 |
---------------- |
| 94 |
It is possible to limit the number of items returned from a collection |
| 95 |
by using either the |
| 96 |
:py:meth:`~boto3.resources.collection.ResourceCollection.limit` method:: |
| 97 |
|
| 98 |
# S3 iterate over first ten buckets |
| 99 |
for bucket in s3.buckets.limit(10): |
| 100 |
print(bucket.name) |
| 101 |
|
| 102 |
In both cases, up to 10 items total will be returned. If you do not |
| 103 |
have 10 buckets, then all of your buckets will be returned. |
| 104 |
|
| 105 |
Controlling page size |
| 106 |
--------------------- |
| 107 |
Collections automatically handle paging through results, but you may want |
| 108 |
to control the number of items returned from a single service operation |
| 109 |
call. You can do so using the |
| 110 |
:py:meth:`~boto3.resources.collection.ResourceCollection.page_size` method:: |
| 111 |
|
| 112 |
# S3 iterate over all objects 100 at a time |
| 113 |
for obj in bucket.objects.page_size(100): |
| 114 |
print(obj.key) |
| 115 |
|
| 116 |
|
| 117 |
By default, S3 will return 1000 objects at a time, so the above code |
| 118 |
would let you process the items in smaller batches, which could be |
| 119 |
beneficial for slow or unreliable internet connections. |
| 120 |
|
| 121 |
Batch actions |
| 122 |
------------- |
| 123 |
Some collections support batch actions, which are actions that operate |
| 124 |
on an entire page of results at a time. They will automatically handle |
| 125 |
pagination:: |
| 126 |
|
| 127 |
# S3 delete everything in `amzn-s3-demo-bucket` |
| 128 |
s3 = boto3.resource('s3') |
| 129 |
s3.Bucket('amzn-s3-demo-bucket').objects.delete() |
| 130 |
|
| 131 |
.. danger:: |
| 132 |
|
| 133 |
The above example will **completely erase all data** in the |
| 134 |
``amzn-s3-demo-bucket`` bucket! Please be careful with batch actions. |