Debian Codegraph @

src/lazr/restfulclient/docs/caching.rst

raw
1 *******
2 Caching
3 *******
4
5 lazr.restfulclient automatically caches the responses to its requests
6 in a temporary directory.
7
8 >>> import httplib2
9 >>> httplib2.debuglevel = 1
10
11 >>> from lazr.restfulclient.tests.example import CookbookWebServiceClient
12 >>> service_with_cache = CookbookWebServiceClient()
13 send: 'GET /1.0/ ...
14 reply: ...200...
15 ...
16 header: Content-Type: application/vnd.sun.wadl+xml
17 ...
18 send: 'GET /1.0/ ...
19 reply: ...200...
20 ...
21 header: Content-Type: application/json
22 ...
23
24 >>> print service_with_cache.recipes[4].instructions
25 send: 'GET /1.0/recipes/4 ...
26 reply: ...200...
27 ...
28 Preheat oven to...
29
30 The second and subsequent times you request some object, it's likely
31 that lazr.restfulclient will make a conditional HTTP GET request instead of
32 a normal request. The HTTP response code will be 304 instead of 200,
33 and lazr.restfulclient will use the cached representation of the object.
34
35 >>> print service_with_cache.recipes[4].instructions
36 send: 'GET /1.0/recipes/4 ...
37 reply: ...304...
38 ...
39 Preheat oven to...
40
41 This is true even if you initially got the object as part of a
42 collection.
43
44 >>> recipes = service_with_cache.recipes[:10]
45 send: ...
46 reply: ...200...
47
48 >>> first_recipe = recipes[0]
49 >>> first_recipe.lp_refresh()
50 send: ...
51 reply: ...304...
52
53 Note that if you get an object as part of a collection and then get it
54 some other way, a conditional GET request will *not* be made. This is
55 a shortcoming of the library.
56
57 >>> service_with_cache.recipes[first_recipe.id]
58 send: ...
59 reply: ...200...
60
61 The default lazr.restfulclient cache directory is a temporary directory
62 that's deleted when the Python process ends. (If the process is
63 killed, the directory will stick around in /tmp.) It's much more
64 efficient to keep a cache directory across multiple uses of
65 lazr.restfulclient.
66
67 You can provide a cache directory name as argument when creating a
68 Service object. This directory will fill up with cached HTTP
69 responses, and since it's a directory you control it will persist
70 across lazr.restfulclient sessions.
71
72 >>> import tempfile
73 >>> tempdir = tempfile.mkdtemp()
74
75 >>> first_service = CookbookWebServiceClient(cache=tempdir)
76 send: 'GET /1.0/ ...
77 reply: ...200...
78 ...
79 send: 'GET /1.0/ ...
80 reply: ...200...
81 ...
82
83 >>> print first_service.recipes[4].instructions
84 send: 'GET /1.0/recipes/4 ...
85 reply: ...200...
86 ...
87 Preheat oven to...
88
89 This will save you a *lot* of time in subsequent sessions, because
90 you'll be able to use cached versions of the initial (very expensive)
91 documents. A new client will not re-request the service root at all.
92
93 >>> second_service = CookbookWebServiceClient(cache=unicode(tempdir))
94
95 You'll also be able to make conditional requests for many resources
96 and avoid transferring their full representations.
97
98 >>> print second_service.recipes[4].instructions
99 send: 'GET /1.0/recipes/4 ...
100 reply: ...304...
101 ...
102 Preheat oven to...
103
104 Of course, if you ever need to clear the cache directory, you'll have
105 to do it yourself.
106
107 Cleanup.
108
109 >>> import shutil
110 >>> shutil.rmtree(tempdir)
111
112 Cache expiration
113 ----------------
114
115 The '1.0' version of the example web service, which we've been using up til
116 now, sets a long cache expiry time for the service root. That's why we
117 were able to create a second client that didn't request the service
118 root at all--just fetched the representations from its cache.
119
120 The 'devel' version of the example web service sets a cache expiry
121 time of two seconds. Let's see what that looks like on the client side.
122
123 >>> tempdir = tempfile.mkdtemp()
124 >>> first_service = CookbookWebServiceClient(
125 ... cache=tempdir, version='devel')
126 send: 'GET /devel/ ...
127 reply: ...200...
128 ...
129 send: 'GET /devel/ ...
130 reply: ...200...
131 ...
132
133 Now let's wait for three seconds to make sure the representations become
134 stale.
135
136 >>> from time import sleep
137 >>> sleep(3)
138
139 When the representations are stale, a new client makes *conditional*
140 requests for the representations. If the conditions fail (as they do
141 here), the cached representations are considered to have been
142 refreshed, just as if the server had sent them again.
143
144 >>> second_service = CookbookWebServiceClient(
145 ... cache=tempdir, version='devel')
146 send: 'GET /devel/ ...
147 reply: ...304...
148 ...
149 send: 'GET /devel/ ...
150 reply: ...304...
151 ...
152
153 Let's quickly create another client before the representation grows
154 stale again.
155
156 >>> second_service = CookbookWebServiceClient(
157 ... cache=tempdir, version='devel')
158
159 When the representations are not stale, a new client does not make any
160 HTTP requests at all--it fetches representations direct from the
161 cache.
162
163 Cleanup.
164
165 >>> httplib2.debuglevel = 0
166 >>> shutil.rmtree(tempdir)
167
168 Cache filenames
169 ---------------
170
171 lazr.restfulclient caches HTTP repsonses in individual files named
172 after the URL accessed. This is behavior derived from httplib2, but
173 lazr.restfulclient does two things differently from httplib2.
174
175 To see these two things, let's set up a client that uses a temporary
176 directory as a cache file. The directory starts out empty.
177
178 >>> from os import listdir
179 >>> tempdir = tempfile.mkdtemp()
180 >>> len(listdir(tempdir))
181 0
182
183 As soon as we create a client object, though, lazr.restfulclient
184 fetches a JSON and a WADL representation of the service root, and
185 caches them individually.
186
187 >>> service = CookbookWebServiceClient(cache=tempdir)
188 >>> cache_contents = listdir(tempdir)
189 >>> for file in sorted(cache_contents):
190 ... print file
191 cookbooks.dev...application,json...
192 cookbooks.dev...vnd.sun.wadl+xml...
193
194 This is the first difference between lazr.restfulclient's caching and
195 httplib2's. httplib2 would store all requests for the service root in
196 a filename based solely on the URL. This effectively limits httplib2
197 to a single representation of a given resource: the WADL
198 representation would be overwritten with the JSON
199 representation. lazr.restfulclient incorporates the media type in the
200 cache filename, so that WADL and JSON representations are stored
201 separately.
202
203 The second difference has to do with filename length limits. httplib2
204 caps filenames at about 240 characters so that cache files can be
205 stored on filesystems with 255-character filename length limits. For
206 compatibility with eCryptfs filesystems, lazr.restfulclient goes
207 further, and caps filenames at 143 characters.
208
209 To test out the limit, let's create a cookbook with an incredibly
210 long name.
211
212 >>> long_name = (
213 ... "This cookbook name is amazingly long; so long that it will "
214 ... "surely be truncated when it is incorporated into a file "
215 ... "name for the cache. The cache file will contain a cached "
216 ... "HTTP respone containing a JSON representation of of this "
217 ... "cookbook, whose name, I repeat, is very long indeed.")
218 >>> len(long_name)
219 281
220
221 >>> import datetime
222 >>> date = datetime.datetime(1994, 1, 1)
223 >>> book = service.cookbooks.create(
224 ... name=long_name, cuisine="General", copyright_date=date,
225 ... price=10.22, last_printing=date)
226
227 lazr.restfulclient automatically fetched a JSON representation of the
228 new cookbook, so it's already present in the cache. Because a
229 cookbook's URL incorporates its name, and this cookbook's name is
230 incredibly long, it must have been truncated to fit on disk.
231
232 >>> [cookbook_cache_filename] = [file for file in listdir(tempdir)
233 ... if 'amazingly' in file]
234
235 Indeed, the filename has been truncated to fit in the rough
236 143-character safety limit for eCryptfs filesystems.
237
238 >>> len(cookbook_cache_filename)
239 143
240
241 Despite the truncation, some of the useful information from the
242 cookbook's name makes it into the filename, making it easy to find when
243 manually crawling through the cache directory.
244
245 >>> print cookbook_cache_filename
246 cookbooks.dev...This%20cookbook%20name%20is%20amazingly%20long...
247
248 To avoid conflicts caused by truncation, the filename always ends with
249 an MD5 sum derived from the untruncated URL. Let's create a second
250 cookbook whose name differs from the first cookbook only at the end.
251
252 >>> longer_name = long_name + ": The Sequel"
253 >>> book = service.cookbooks.create(
254 ... name=longer_name, cuisine="General", copyright_date=date,
255 ... price=10.22, last_printing=date)
256
257 This cookbook's URL is identical to the first cookbook's URL for far
258 longer than 143 characters. But since the truncated filename
259 incorporates an MD5 sum based on the full URL, the two cookbooks are
260 cached in separate files.
261
262 >>> [file1, file2] = [file for file in listdir(tempdir)
263 ... if 'amazingly' in file]
264
265 The filenames are identical up to the last 32 characters, which is
266 where the MD5 sum begins. But because the MD5 sums are different, they
267 are not completely identical.
268
269 >>> file1[:-32] == file2[:-32]
270 True
271
272 >>> file1 == file2
273 False
274
275 Cleanup.
276
277 >>> import shutil
278 >>> shutil.rmtree(tempdir)