Debian Codegraph @

docs/source/guide/paginators.rst

raw
1 Paginators
2 ==========
3
4 Some AWS operations return results that are incomplete and require subsequent
5 requests in order to attain the entire result set. The process of sending
6 subsequent requests to continue where a previous request left off is called
7 *pagination*. For example, the ``list_objects`` operation of Amazon S3
8 returns up to 1000 objects at a time, and you must send subsequent requests
9 with the appropriate ``Marker`` in order to retrieve the next *page* of
10 results.
11
12 *Paginators* are a feature of boto3 that act as an abstraction over the
13 process of iterating over an entire result set of a truncated API operation.
14
15
16 Creating paginators
17 -------------------
18
19 Paginators are created via the ``get_paginator()`` method of a boto3
20 client. The ``get_paginator()`` method accepts an operation name and returns
21 a reusable ``Paginator`` object. You then call the ``paginate`` method of the
22 Paginator, passing in any relevant operation parameters to apply to the
23 underlying API operation. The ``paginate`` method then returns an iterable
24 ``PageIterator``::
25
26 import boto3
27
28 # Create a client
29 client = boto3.client('s3', region_name='us-west-2')
30
31 # Create a reusable Paginator
32 paginator = client.get_paginator('list_objects_v2')
33
34 # Create a PageIterator from the Paginator
35 page_iterator = paginator.paginate(Bucket='amzn-s3-demo-bucket')
36
37 for page in page_iterator:
38 print(page['Contents'])
39
40
41 Customizing page iterators
42 ~~~~~~~~~~~~~~~~~~~~~~~~~~
43
44 You must call the ``paginate`` method of a Paginator in order to iterate over
45 the pages of API operation results. The ``paginate`` method accepts a
46 ``PaginationConfig`` named argument that can be used to customize the
47 pagination::
48
49 paginator = client.get_paginator('list_objects_v2')
50 page_iterator = paginator.paginate(Bucket='amzn-s3-demo-bucket',
51 PaginationConfig={'MaxItems': 10})
52
53 ``MaxItems``
54 Limits the maximum number of total returned items returned while
55 paginating.
56
57 ``StartingToken``
58 Can be used to modify the starting marker or token of a paginator. This
59 argument if useful for resuming pagination from a previous token or
60 starting pagination at a known position.
61
62 ``PageSize``
63 Controls the number of items returned per page of each result.
64
65 .. note::
66
67 Services may choose to return more or fewer items than specified in the
68 ``PageSize`` argument depending on the service, the operation, or the
69 resource you are paginating.
70
71
72 Filtering results
73 -----------------
74
75 Many Paginators can be filtered server-side with options that are passed
76 through to each underlying API call. For example,
77 :py:meth:`S3.Paginator.list_objects.paginate` accepts a ``Prefix`` parameter
78 used to filter the paginated results by prefix server-side before sending them
79 to the client::
80
81 import boto3
82
83 client = boto3.client('s3', region_name='us-west-2')
84 paginator = client.get_paginator('list_objects_v2')
85 operation_parameters = {'Bucket': 'amzn-s3-demo-bucket',
86 'Prefix': 'foo/baz'}
87 page_iterator = paginator.paginate(**operation_parameters)
88 for page in page_iterator:
89 print(page['Contents'])
90
91
92 Filtering results with JMESPath
93 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
94
95 `JMESPath <http://jmespath.org>`_ is a query language for JSON that can be used
96 directly on paginated results. You can filter results client-side using
97 JMESPath expressions that are applied to each page of results through the
98 ``search`` method of a ``PageIterator``.
99
100 .. code-block:: python
101
102 import boto3
103
104 client = boto3.client('s3', region_name='us-west-2')
105 paginator = client.get_paginator('list_objects_v2')
106 page_iterator = paginator.paginate(Bucket='amzn-s3-demo-bucket')
107 filtered_iterator = page_iterator.search("Contents[?Size > `100`][]")
108 for key_data in filtered_iterator:
109 print(key_data)
110
111 When filtering with JMESPath expressions, each page of results that is yielded
112 by the paginator is mapped through the JMESPath expression. If a JMESPath
113 expression returns a single value that is not an array, that value is yielded
114 directly. If the result of applying the JMESPath expression to a page of
115 results is a list, then each value of the list is yielded individually
116 (essentially implementing a flat map). For example, in the above expression,
117 each key that has a ``Size`` greater than `100` is yielded by the
118 ``filtered_iterator``.