| 1 |
Retry requests on server error |
| 2 |
****************************** |
| 3 |
|
| 4 |
If lazr.restfulclient talks to a server that sends out a server-side |
| 5 |
error with status codes 502 or 503, the client will wait a few seconds |
| 6 |
and try the request again. Eventually it will give up and escalate the |
| 7 |
error code in the form of an exception. |
| 8 |
|
| 9 |
To test this, let's simulate a lazr.restful server prone to transient |
| 10 |
errors using a WSGI application. |
| 11 |
|
| 12 |
>>> import pkg_resources |
| 13 |
>>> wadl_string = pkg_resources.resource_string( |
| 14 |
... 'wadllib.tests.data', 'launchpad-wadl.xml') |
| 15 |
>>> representations = { 'application/vnd.sun.wadl+xml' : wadl_string, |
| 16 |
... 'application/json' : '{}' } |
| 17 |
|
| 18 |
This application will cause one request to fail for every item in its |
| 19 |
BROKEN_RESPONSES list. |
| 20 |
|
| 21 |
>>> BROKEN_RESPONSES = [] |
| 22 |
>>> def broken_application(environ, start_response): |
| 23 |
... if len(BROKEN_RESPONSES) > 0: |
| 24 |
... start_response(str(BROKEN_RESPONSES.pop()), |
| 25 |
... [('Content-type', 'text/plain')]) |
| 26 |
... return ["Sorry, I'm still broken."] |
| 27 |
... else: |
| 28 |
... media_type = environ['HTTP_ACCEPT'] |
| 29 |
... content = representations[media_type] |
| 30 |
... start_response( |
| 31 |
... '200', [('Content-type', media_type)]) |
| 32 |
... return [content] |
| 33 |
|
| 34 |
>>> def make_broken_application(): |
| 35 |
... return broken_application |
| 36 |
|
| 37 |
>>> import wsgi_intercept |
| 38 |
>>> wsgi_intercept.add_wsgi_intercept( |
| 39 |
... 'api.launchpad.dev', 80, make_broken_application) |
| 40 |
>>> BROKEN_RESPONSES = [] |
| 41 |
|
| 42 |
>>> from wsgi_intercept.httplib2_intercept import install |
| 43 |
>>> install() |
| 44 |
|
| 45 |
Here's a fake implementation of time.sleep() so that this test doesn't |
| 46 |
take a really long time to run, and so we can visualize sleep() being |
| 47 |
called as lazr.restfulclient retries over and over again. |
| 48 |
|
| 49 |
>>> def fake_sleep(time): |
| 50 |
... print "sleep(%s) called" % time |
| 51 |
>>> import lazr.restfulclient._browser |
| 52 |
>>> old_sleep = lazr.restfulclient._browser.sleep |
| 53 |
>>> lazr.restfulclient._browser.sleep = fake_sleep |
| 54 |
|
| 55 |
As it starts out, the application isn't broken at all. |
| 56 |
|
| 57 |
>>> from lazr.restfulclient.resource import ServiceRoot |
| 58 |
>>> client = ServiceRoot(None, "http://api.launchpad.dev/") |
| 59 |
|
| 60 |
Let's queue up one broken response. The client will sleep once and |
| 61 |
try again. |
| 62 |
|
| 63 |
>>> BROKEN_RESPONSES = [502] |
| 64 |
>>> client = ServiceRoot(None, "http://api.launchpad.dev/") |
| 65 |
sleep(0) called |
| 66 |
|
| 67 |
Now the application will fail six times and then start working. |
| 68 |
|
| 69 |
>>> BROKEN_RESPONSES = [502, 503, 502, 503, 502, 503] |
| 70 |
>>> client = ServiceRoot(None, "http://api.launchpad.dev/") |
| 71 |
sleep(0) called |
| 72 |
sleep(1) called |
| 73 |
sleep(2) called |
| 74 |
sleep(4) called |
| 75 |
sleep(8) called |
| 76 |
sleep(16) called |
| 77 |
|
| 78 |
Now the application will fail seven times and then start working. But |
| 79 |
the client will give up before then--it will only retry the request |
| 80 |
six times. |
| 81 |
|
| 82 |
>>> BROKEN_RESPONSES = [502, 503, 502, 503, 502, 503, 502] |
| 83 |
>>> client = ServiceRoot(None, "http://api.launchpad.dev/") |
| 84 |
Traceback (most recent call last): |
| 85 |
... |
| 86 |
ServerError: HTTP Error 502: |
| 87 |
... |
| 88 |
|
| 89 |
By increasing the 'max_retries' constructor argument, we can make the |
| 90 |
application try more than six times, and eventually succeed. |
| 91 |
|
| 92 |
>>> BROKEN_RESPONSES = [502, 503, 502, 503, 502, 503, 502] |
| 93 |
>>> client = ServiceRoot(None, "http://api.launchpad.dev/", |
| 94 |
... max_retries=10) |
| 95 |
sleep(0) called |
| 96 |
sleep(1) called |
| 97 |
sleep(2) called |
| 98 |
sleep(4) called |
| 99 |
sleep(8) called |
| 100 |
sleep(16) called |
| 101 |
sleep(32) called |
| 102 |
|
| 103 |
Now the application will fail once and then give a 400 error. The |
| 104 |
client will not retry in hopes that the 400 error will go away--400 is |
| 105 |
a client error. |
| 106 |
|
| 107 |
>>> BROKEN_RESPONSES = [502, 400] |
| 108 |
>>> client = ServiceRoot(None, "http://api.launchpad.dev/") |
| 109 |
Traceback (most recent call last): |
| 110 |
... |
| 111 |
BadRequest: HTTP Error 400: |
| 112 |
... |
| 113 |
|
| 114 |
Teardown. |
| 115 |
|
| 116 |
>>> _ = wsgi_intercept.remove_wsgi_intercept("api.launchpad.dev", 80) |
| 117 |
>>> lazr.restfulclient._browser.sleep = old_sleep |