Debian Codegraph @

docs/source/guide/retries.rst

raw
1 .. _guide_retries:
2
3 Retries
4 =======
5
6 Overview
7 --------
8
9 Your AWS client might see calls to AWS services fail due to unexpected issues on the client side. Or calls might fail due to rate limiting from the AWS service you're attempting to call. In either case, these kinds of failures often don’t require special handling and the call should be made again, often after a brief waiting period. Boto3 provides many features to assist in retrying client calls to AWS services when these kinds of errors or exceptions are experienced.
10
11 This guide provides you with details on the following:
12
13 * How to find the available retry modes and the differences between each mode
14 * How to configure your client to use each retry mode and other retry configurations
15 * How to validate if your client performs a retry attempt
16
17 Available retry modes
18 ---------------------
19
20 Legacy retry mode
21 ~~~~~~~~~~~~~~~~~~
22
23 Legacy mode is the default mode used by any Boto3 client you create. As its name implies, ``legacy mode`` uses an older (v1) retry handler that has limited functionality.
24
25 **Legacy mode’s functionality includes:**
26
27 * A default value of 5 for maximum attempts (including the initial request). See `Available configuration options`_ for more information on overwriting this value.
28 * Retry attempts for a limited number of errors/exceptions::
29
30 # General socket/connection errors
31 ConnectionError
32 ConnectionClosedError
33 ReadTimeoutError
34 EndpointConnectionError
35
36 # Service-side throttling/limit errors and exceptions
37 Throttling
38 ThrottlingException
39 ThrottledException
40 RequestThrottledException
41 ProvisionedThroughputExceededException
42
43 * Retry attempts on several HTTP status codes, including 429, 500, 502, 503, 504, and 509.
44 * Any retry attempt will include an exponential backoff by a base factor of 2.
45
46
47 .. note::
48 For more information about additional service-specific retry policies, see the following `botocore references in GitHub <https://github.com/boto/botocore/blob/develop/botocore/data/_retry.json>`_.
49
50
51 Standard retry mode
52 ~~~~~~~~~~~~~~~~~~~~
53
54 Standard mode is a retry mode that was introduced with the updated retry handler (v2). This mode is a standardization of retry logic and behavior that is consistent with other AWS SDKs. In addition to this standardization, this mode also extends the functionality of retries over that found in legacy mode.
55
56 **Standard mode’s functionality includes:**
57
58 * A default value of 3 for maximum attempts (including the initial request). See `Available configuration options`_ for more information on overwriting this value.
59 * Circuit-breaking functionality to prevent retry attempts during service outages.
60 * Retry attempts for an expanded list of errors/exceptions::
61
62 # Transient errors/exceptions
63 RequestTimeout
64 RequestTimeoutException
65 PriorRequestNotComplete
66 ConnectionError
67 HTTPClientError
68
69 # Service-side throttling/limit errors and exceptions
70 Throttling
71 ThrottlingException
72 ThrottledException
73 RequestThrottledException
74 TooManyRequestsException
75 ProvisionedThroughputExceededException
76 TransactionInProgressException
77 RequestLimitExceeded
78 BandwidthLimitExceeded
79 LimitExceededException
80 RequestThrottled
81 SlowDown
82 EC2ThrottledException
83
84 * Retry attempts on nondescriptive, transient error codes. Specifically, these HTTP status codes: 500, 502, 503, 504.
85 * Any retry attempt will include an exponential backoff by a base factor of 2 for a maximum backoff time of 20 seconds.
86
87 Adaptive retry mode
88 ~~~~~~~~~~~~~~~~~~~~
89
90 Adaptive retry mode is an experimental retry mode that includes all the features of standard mode. In addition to the standard mode features, adaptive mode also introduces client-side rate limiting through the use of a token bucket and rate-limit variables that are dynamically updated with each retry attempt. This mode offers flexibility in client-side retries that adapts to the error/exception state response from an AWS service.
91
92 With each new retry attempt, adaptive mode modifies the rate-limit variables based on the error, exception, or HTTP status code presented in the response from the AWS service. These rate-limit variables are then used to calculate a new call rate for the client. Each exception/error or non-success HTTP response (provided in the list above) from an AWS service updates the rate-limit variables as retries occur until success is reached, the token bucket is exhausted, or the configured maximum attempts value is reached.
93
94 .. note::
95 Adaptive mode is an experimental mode and is subject to change, both in features and behavior.
96
97
98 Configuring a retry mode
99 -------------------------
100
101 Boto3 includes a variety of both retry configurations as well as configuration methods to consider when creating your client object.
102
103 Available configuration options
104 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
105
106 In Boto3, users can customize retry configurations:
107
108 * ``retry_mode`` - This tells Boto3 which retry mode to use. As described previously, there are three retry modes available: legacy (default), standard, and adaptive.
109 * ``max_attempts`` - This provides Boto3's retry handler with a value of maximum attempts. **Important**: The behavior differs depending on how it's configured:
110
111 * When set in your AWS config file or using the ``AWS_MAX_ATTEMPTS`` environment variable: ``max_attempts`` includes the initial request (total requests)
112 * When set in a ``Config`` object: ``max_attempts`` excludes the initial request (retries only)
113
114 **Examples:**
115
116 * AWS config file with ``max_attempts = 3``: 1 initial request + 2 retries = 3 total attempts
117 * Environment variable ``AWS_MAX_ATTEMPTS=3``: 1 initial request + 2 retries = 3 total attempts
118 * Config object with ``max_attempts: 3``: 1 initial request + 3 retries = 4 total attempts
119
120 * ``total_max_attempts`` - Available only in ``Config`` objects, this always represents total requests including the initial call. This parameter was introduced to provide consistent behavior with the ``max_attempts`` setting used in AWS config files and environment variables. Note that ``total_max_attempts`` is not supported as an environment variable or in AWS config files.
121
122 For consistency, consider using ``total_max_attempts`` in ``Config`` objects instead of ``max_attempts``.
123
124 Defining a retry configuration in your AWS configuration file
125 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
126
127 This first way to define your retry configuration is to update your global AWS configuration file. The default location for your AWS config file is ``~/.aws/config``. Here’s an example of an AWS config file with the retry configuration options used::
128
129 [myConfigProfile]
130 region = us-east-1
131 max_attempts = 10
132 retry_mode = standard
133
134 Any Boto3 script or code that uses your AWS config file inherits these configurations when using your profile, unless otherwise explicitly overwritten by a ``Config`` object when instantiating your client object at runtime. If no configuration options are set, the default retry mode value is ``legacy``, and the default ``max_attempts`` value is 5 (total attempts including initial request).
135
136 Defining a retry configuration in a Config object for your Boto3 client
137 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
138
139 The second way to define your retry configuration is to use botocore to enable more flexibility for you to specify your retry configuration using a ``Config`` object that you can pass to your client at runtime. This method is useful if you don't want to configure retry behavior globally with your AWS config file
140
141 Additionally, if your AWS configuration file is configured with retry behavior, but you want to override those global settings, you can use the ``Config`` object to override an individual client object at runtime.
142
143 As shown in the following example, the ``Config`` object takes a ``retries`` dictionary where you can supply configuration options such as ``total_max_attempts`` and ``mode``, and the values for each.
144
145 .. code-block:: python
146
147 config = Config(
148 retries = {
149 'total_max_attempts': 10,
150 'mode': 'standard'
151 }
152 )
153
154 .. note::
155 The AWS configuration file uses ``retry_mode`` and the ``Config`` object uses ``mode``. Although named differently, they both refer to the same retry configuration whose options are legacy (default), standard, and adaptive.
156
157 The following is an example of instantiating a ``Config`` object and passing it into an Amazon EC2 client to use at runtime.
158
159 .. code-block:: python
160
161 import boto3
162 from botocore.config import Config
163
164 config = Config(
165 retries = {
166 'total_max_attempts': 10,
167 'mode': 'standard'
168 }
169 )
170
171 ec2 = boto3.client('ec2', config=config)
172
173 .. note::
174 As mentioned previously, if no configuration options are set, the default mode is ``legacy`` and the default ``total_max_attempts`` is 5 (total attempts including initial request).
175
176
177 Validating retry attempts
178 --------------------------
179
180 To ensure that your retry configuration is correct and working properly, there are a number of ways you can validate that your client's retries are occurring.
181
182 Checking retry attempts in your client logs
183 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
184
185 If you enable Boto3’s logging, you can validate and check your client’s retry attempts in your client’s logs. Notice, however, that you need to enable ``DEBUG`` mode in your logger to see any retry attempts. The client log entries for retry attempts will appear differently, depending on which retry mode you’ve configured.
186
187 **If legacy mode is enabled:**
188
189 Retry messages are generated by ``botocore.retryhandler``. You’ll see one of three messages:
190
191 * *No retry needed*
192 * *Retry needed, action of: <action_value>*
193 * *Reached the maximum number of retry attempts: <attempt_num>*
194
195
196 **If standard or adaptive mode is enabled:**
197
198 Retry messages are generated by ``botocore.retries.standard``. You’ll see one of three messages:
199
200 * *Not retrying request*
201 * *Retry needed, retrying request after delay of: <delay_value>*
202 * *Retry needed but retry quota reached, not retrying request*
203
204 Checking retry attempts in an AWS service response
205 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
206
207 You can check the number of retry attempts your client has made by parsing the response botocore provides when making a call to an AWS service API. Responses are handled by an underlying botocore module, and formatted into a dictionary that's part of the JSON response object. You can access the number of retry attempts your client has taken by calling the ``RetryAttempts`` key in the ``ResponseMetaData`` dictionary::
208
209 'ResponseMetadata': {
210 'RequestId': '1234567890ABCDEF',
211 'HostId': 'host ID data will appear here as a hash',
212 'HTTPStatusCode': 400,
213 'HTTPHeaders': {'header metadata key/values will appear here'},
214 'RetryAttempts': 4
215 }