| 1 |
Paginators |
| 2 |
========== |
| 3 |
|
| 4 |
Some AWS operations return results that are incomplete and require subsequent |
| 5 |
requests in order to attain the entire result set. The process of sending |
| 6 |
subsequent requests to continue where a previous request left off is called |
| 7 |
*pagination*. For example, the ``list_objects`` operation of Amazon S3 |
| 8 |
returns up to 1000 objects at a time, and you must send subsequent requests |
| 9 |
with the appropriate ``Marker`` in order to retrieve the next *page* of |
| 10 |
results. |
| 11 |
|
| 12 |
*Paginators* are a feature of boto3 that act as an abstraction over the |
| 13 |
process of iterating over an entire result set of a truncated API operation. |
| 14 |
|
| 15 |
|
| 16 |
Creating paginators |
| 17 |
------------------- |
| 18 |
|
| 19 |
Paginators are created via the ``get_paginator()`` method of a boto3 |
| 20 |
client. The ``get_paginator()`` method accepts an operation name and returns |
| 21 |
a reusable ``Paginator`` object. You then call the ``paginate`` method of the |
| 22 |
Paginator, passing in any relevant operation parameters to apply to the |
| 23 |
underlying API operation. The ``paginate`` method then returns an iterable |
| 24 |
``PageIterator``:: |
| 25 |
|
| 26 |
import boto3 |
| 27 |
|
| 28 |
# Create a client |
| 29 |
client = boto3.client('s3', region_name='us-west-2') |
| 30 |
|
| 31 |
# Create a reusable Paginator |
| 32 |
paginator = client.get_paginator('list_objects_v2') |
| 33 |
|
| 34 |
# Create a PageIterator from the Paginator |
| 35 |
page_iterator = paginator.paginate(Bucket='amzn-s3-demo-bucket') |
| 36 |
|
| 37 |
for page in page_iterator: |
| 38 |
print(page['Contents']) |
| 39 |
|
| 40 |
|
| 41 |
Customizing page iterators |
| 42 |
~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 43 |
|
| 44 |
You must call the ``paginate`` method of a Paginator in order to iterate over |
| 45 |
the pages of API operation results. The ``paginate`` method accepts a |
| 46 |
``PaginationConfig`` named argument that can be used to customize the |
| 47 |
pagination:: |
| 48 |
|
| 49 |
paginator = client.get_paginator('list_objects_v2') |
| 50 |
page_iterator = paginator.paginate(Bucket='amzn-s3-demo-bucket', |
| 51 |
PaginationConfig={'MaxItems': 10}) |
| 52 |
|
| 53 |
``MaxItems`` |
| 54 |
Limits the maximum number of total returned items returned while |
| 55 |
paginating. |
| 56 |
|
| 57 |
``StartingToken`` |
| 58 |
Can be used to modify the starting marker or token of a paginator. This |
| 59 |
argument if useful for resuming pagination from a previous token or |
| 60 |
starting pagination at a known position. |
| 61 |
|
| 62 |
``PageSize`` |
| 63 |
Controls the number of items returned per page of each result. |
| 64 |
|
| 65 |
.. note:: |
| 66 |
|
| 67 |
Services may choose to return more or fewer items than specified in the |
| 68 |
``PageSize`` argument depending on the service, the operation, or the |
| 69 |
resource you are paginating. |
| 70 |
|
| 71 |
|
| 72 |
Filtering results |
| 73 |
----------------- |
| 74 |
|
| 75 |
Many Paginators can be filtered server-side with options that are passed |
| 76 |
through to each underlying API call. For example, |
| 77 |
:py:meth:`S3.Paginator.list_objects.paginate` accepts a ``Prefix`` parameter |
| 78 |
used to filter the paginated results by prefix server-side before sending them |
| 79 |
to the client:: |
| 80 |
|
| 81 |
import boto3 |
| 82 |
|
| 83 |
client = boto3.client('s3', region_name='us-west-2') |
| 84 |
paginator = client.get_paginator('list_objects_v2') |
| 85 |
operation_parameters = {'Bucket': 'amzn-s3-demo-bucket', |
| 86 |
'Prefix': 'foo/baz'} |
| 87 |
page_iterator = paginator.paginate(**operation_parameters) |
| 88 |
for page in page_iterator: |
| 89 |
print(page['Contents']) |
| 90 |
|
| 91 |
|
| 92 |
Filtering results with JMESPath |
| 93 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 94 |
|
| 95 |
`JMESPath <http://jmespath.org>`_ is a query language for JSON that can be used |
| 96 |
directly on paginated results. You can filter results client-side using |
| 97 |
JMESPath expressions that are applied to each page of results through the |
| 98 |
``search`` method of a ``PageIterator``. |
| 99 |
|
| 100 |
.. code-block:: python |
| 101 |
|
| 102 |
import boto3 |
| 103 |
|
| 104 |
client = boto3.client('s3', region_name='us-west-2') |
| 105 |
paginator = client.get_paginator('list_objects_v2') |
| 106 |
page_iterator = paginator.paginate(Bucket='amzn-s3-demo-bucket') |
| 107 |
filtered_iterator = page_iterator.search("Contents[?Size > `100`][]") |
| 108 |
for key_data in filtered_iterator: |
| 109 |
print(key_data) |
| 110 |
|
| 111 |
When filtering with JMESPath expressions, each page of results that is yielded |
| 112 |
by the paginator is mapped through the JMESPath expression. If a JMESPath |
| 113 |
expression returns a single value that is not an array, that value is yielded |
| 114 |
directly. If the result of applying the JMESPath expression to a page of |
| 115 |
results is a list, then each value of the list is yielded individually |
| 116 |
(essentially implementing a flat map). For example, in the above expression, |
| 117 |
each key that has a ``Size`` greater than `100` is yielded by the |
| 118 |
``filtered_iterator``. |