Debian Codegraph @

docs/source/guide/collections.rst

raw
1 .. _guide_collections:
2
3 Collections
4 ===========
5
6 Overview
7 --------
8 A collection provides an iterable interface to a group of resources.
9 Collections behave similarly to
10 `Django QuerySets <https://docs.djangoproject.com/en/4.2/ref/models/querysets/>`_
11 and expose a similar API. A collection seamlessly handles pagination for
12 you, making it possible to easily iterate over all items from all pages of
13 data. Example of a collection::
14
15 # SQS list all queues
16 sqs = boto3.resource('sqs')
17 for queue in sqs.queues.all():
18 print(queue.url)
19
20 When collections make requests
21 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
22 Collections can be created and manipulated without any request being made
23 to the underlying service. A collection makes a remote service request under
24 the following conditions:
25
26 * **Iteration**::
27
28 for bucket in s3.buckets.all():
29 print(bucket.name)
30
31 * **Conversion to list()**::
32
33 buckets = list(s3.buckets.all())
34
35 * **Batch actions (see below)**::
36
37 s3.Bucket('amzn-s3-demo-bucket').objects.delete()
38
39 Filtering
40 ---------
41 Some collections support extra arguments to filter the returned data set,
42 which are passed into the underlying service operation. Use the
43 :py:meth:`~boto3.resources.collection.Collection.filter` method to filter
44 the results::
45
46 # S3 list all keys with the prefix 'photos/'
47 s3 = boto3.resource('s3')
48 for bucket in s3.buckets.all():
49 for obj in bucket.objects.filter(Prefix='photos/'):
50 print('{0}:{1}'.format(bucket.name, obj.key))
51
52 .. warning::
53
54 Behind the scenes, the above example will call ``ListBuckets``,
55 ``ListObjects``, and ``HeadObject`` many times. If you have a large
56 number of S3 objects then this could incur a significant cost.
57
58 Chainability
59 ------------
60 Collection methods are chainable. They return copies of the collection
61 rather than modifying the collection, including a deep copy of any
62 associated operation parameters. For example, this allows you
63 to build up multiple collections from a base which they all have
64 in common::
65
66 # EC2 find instances
67 ec2 = boto3.resource('ec2')
68 base = ec2.instances.filter(InstanceIds=['id1', 'id2', 'id3'])
69
70 filters = [{
71 'Name': 'tenancy',
72 'Values': ['dedicated']
73 }]
74 filtered1 = base.filter(Filters=filters)
75
76 # Note, this does NOT modify the filters in ``filtered1``!
77 filters.append({'name': 'instance-type', 'value': 't1.micro'})
78 filtered2 = base.filter(Filters=filters)
79
80 print('All instances:')
81 for instance in base:
82 print(instance.id)
83
84 print('Dedicated instances:')
85 for instance in filtered1:
86 print(instance.id)
87
88 print('Dedicated micro instances:')
89 for instance in filtered2:
90 print(instance.id)
91
92 Limiting results
93 ----------------
94 It is possible to limit the number of items returned from a collection
95 by using either the
96 :py:meth:`~boto3.resources.collection.ResourceCollection.limit` method::
97
98 # S3 iterate over first ten buckets
99 for bucket in s3.buckets.limit(10):
100 print(bucket.name)
101
102 In both cases, up to 10 items total will be returned. If you do not
103 have 10 buckets, then all of your buckets will be returned.
104
105 Controlling page size
106 ---------------------
107 Collections automatically handle paging through results, but you may want
108 to control the number of items returned from a single service operation
109 call. You can do so using the
110 :py:meth:`~boto3.resources.collection.ResourceCollection.page_size` method::
111
112 # S3 iterate over all objects 100 at a time
113 for obj in bucket.objects.page_size(100):
114 print(obj.key)
115
116
117 By default, S3 will return 1000 objects at a time, so the above code
118 would let you process the items in smaller batches, which could be
119 beneficial for slow or unreliable internet connections.
120
121 Batch actions
122 -------------
123 Some collections support batch actions, which are actions that operate
124 on an entire page of results at a time. They will automatically handle
125 pagination::
126
127 # S3 delete everything in `amzn-s3-demo-bucket`
128 s3 = boto3.resource('s3')
129 s3.Bucket('amzn-s3-demo-bucket').objects.delete()
130
131 .. danger::
132
133 The above example will **completely erase all data** in the
134 ``amzn-s3-demo-bucket`` bucket! Please be careful with batch actions.