lazr.restful

Merge lp:~leonardr/lazr.restful/cache-service-root into lp:lazr.restful

  1. cache-service-root
  2. Merge into trunk
Proposed by Leonard Richardson
Status: Merged
Approved by: Gavin Panella
Approved revision: 126
Merged at revision: not available
Proposed branch: lp:~leonardr/lazr.restful/cache-service-root
Merge into: lp:lazr.restful
Diff against target: 217 lines (+145/-2)
5 files modified
src/lazr/restful/NEWS.txt (+12/-0)
src/lazr/restful/_resource.py (+30/-2)
src/lazr/restful/example/base/root.py (+1/-0)
src/lazr/restful/example/base/tests/root.txt (+88/-0)
src/lazr/restful/interfaces/_rest.py (+14/-0)
To merge this branch: bzr merge lp:~leonardr/lazr.restful/cache-service-root
Reviewer Review Type Date Requested Status
Gavin Panella Approve
Review via email: mp+22948@code.launchpad.net

Description of the change

This branch gets lazr.restful to set the Cache-Control header when serving a representation of the service root. This is a big win because the service root only changes when you deploy a new version of your lazr.restful application (eg. Launchpad), but every user makes 2 requests for it every time they start up lazr.restfulclient.

There are two 'max-age' times set in the Cache-Control header: one for the latest version of the web service (which changes more often) and one for all other versions. By default, the max-age for the latest version is one hour and the max-age for all other versions is one week.

After the max-age expires, the client will make a *conditional* request to see whether the service root has changed. If it hasn't, I believe the client will update the client-side max-age and not make any requests for another hour (or week). But I need to check to make sure httplib2 actually does this.

To post a comment you must log in.
Revision history for this message
Gavin Panella (allenap) wrote :

Nice branch. Conversation from IRC:

<allenap> leonardr: Is there any way config.active_versions could be empty?
<leonardr> allenap: no, that would prevent lazr.restful from starting up
<allenap> leonardr: Instead of having [604800, 3600], could you have a module-level constant like `HOUR = 3600 # seconds`, then the default can be specified as [7 * 24 * HOUR, 1 * HOUR]?
<leonardr> sure

review: Approve
lp:~leonardr/lazr.restful/cache-service-root updated
127. By Leonard Richardson

Set the Date header as well, since httplib2 uses it to determine whether a cached representation is stale.

Revision history for this message
Leonard Richardson (leonardr) wrote :

In my experiments with lazr.restfulclient I discovered that httplib2 will ignore Cache-Control unless the Date header is also set to provide a starting point. I've updated the branch to set the Date header.

Revision history for this message
Gavin Panella (allenap) wrote :

The late addition, r127, looks good too.

review: Approve
lp:~leonardr/lazr.restful/cache-service-root updated
128. By Leonard Richardson

Set Cache-Control and Date even on a conditional request.

129. By Leonard Richardson

Set caching headers whether the request was conditional or not.

Revision history for this message
Leonard Richardson (leonardr) wrote :

UNfortunately, this branch breaks lazr.restfulclient due to what I believe is a bug in httplib2 (http://code.google.com/p/httplib2/issues/detail?id=97). We can still land this branch, but we'll only be able to serve the goodness to clients that signal they can work around the bug. This means changing clients to send custom values for User-Agent.

lp:~leonardr/lazr.restful/cache-service-root updated
130. By Leonard Richardson

Don't serve cache control headers to httplib2 clients to avoid triggering a bug in httplib2.

131. By Leonard Richardson

Removed temporary hack for testing purposes.

Revision history for this message
Leonard Richardson (leonardr) wrote :

All right, I've changed lazr.restful to avoid triggering the httplib2 bug on old versions of lazr.restfulclient. (I've tested this version of lazr.restful with an old and a new lazr.restfulclient, and they both worked, though obviously the old one didn't get the benefit of the Cache-Control headers.)

lp:~leonardr/lazr.restful/cache-service-root updated
132. By Leonard Richardson

Slight refactoring.

Revision history for this message
Gavin Panella (allenap) wrote :

Still looking good :)

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'src/lazr/restful/NEWS.txt'
--- src/lazr/restful/NEWS.txt 2010-03-17 14:38:16 +0000
+++ src/lazr/restful/NEWS.txt 2010-04-13 15:37:34 +0000
@@ -2,6 +2,18 @@
2NEWS for lazr.restful2NEWS for lazr.restful
3=====================3=====================
44
5Development
6===========
7
8Special note: This version introduces a new configuration element,
9'caching_policy'. This element starts out simple but may become more
10complex in future versions. See the IWebServiceConfiguration interface
11for more details.
12
13Service root resources are now client-side cacheable for an amount of
14time that depends on the server configuration and the version of the
15web service requested.
16
50.9.24 (2010-03-17)170.9.24 (2010-03-17)
6====================18====================
719
820
=== modified file 'src/lazr/restful/_resource.py'
--- src/lazr/restful/_resource.py 2010-03-16 15:07:00 +0000
+++ src/lazr/restful/_resource.py 2010-04-13 15:37:34 +0000
@@ -32,9 +32,11 @@
32import copy32import copy
33from cStringIO import StringIO33from cStringIO import StringIO
34from datetime import datetime, date34from datetime import datetime, date
35from email.Utils import formatdate
35from gzip import GzipFile36from gzip import GzipFile
36import os37import os
37import simplejson38import simplejson
39import time
38import zlib40import zlib
3941
40# Import SHA in a way compatible with both Python 2.4 and Python 2.6.42# Import SHA in a way compatible with both Python 2.4 and Python 2.6.
@@ -1621,10 +1623,36 @@
1621 result = ""1623 result = ""
1622 return self.applyTransferEncoding(result)1624 return self.applyTransferEncoding(result)
16231625
1626 def setCachingHeaders(self):
1627 "How long should the client cache this service root?"
1628 user_agent = self.request.getHeader('User-Agent', '')
1629 if user_agent.startswith('Python-httplib2'):
1630 # XXX leonardr 20100412
1631 # bug=http://code.google.com/p/httplib2/issues/detail?id=97
1632 #
1633 # A client with a User-Agent of "Python/httplib2" (such as
1634 # old versions of lazr.restfulclient) gives inconsistent
1635 # results when a resource is served with both ETag and
1636 # Cache-Control. We check for that User-Agent and omit the
1637 # Cache-Control headers if it makes a request.
1638 return
1639 config = getUtility(IWebServiceConfiguration)
1640 caching_policy = config.caching_policy
1641 if self.request.version == config.active_versions[-1]:
1642 max_age = caching_policy[-1]
1643 else:
1644 max_age = caching_policy[0]
1645 if max_age > 0:
1646 self.request.response.setHeader(
1647 'Cache-Control', 'max-age=%d' % max_age)
1648 # Also set the Date header so that client-side caches will
1649 # have something to work from.
1650 self.request.response.setHeader('Date', formatdate(time.time()))
1651
1624 def do_GET(self):1652 def do_GET(self):
1625 """Describe the capabilities of the web service in WADL."""1653 """Describe the capabilities of the web service."""
1626
1627 media_type = self.handleConditionalGET()1654 media_type = self.handleConditionalGET()
1655 self.setCachingHeaders()
1628 if media_type is None:1656 if media_type is None:
1629 # The conditional GET succeeded. Serve nothing.1657 # The conditional GET succeeded. Serve nothing.
1630 return ""1658 return ""
16311659
=== modified file 'src/lazr/restful/example/base/root.py'
--- src/lazr/restful/example/base/root.py 2010-03-10 18:45:04 +0000
+++ src/lazr/restful/example/base/root.py 2010-04-13 15:37:34 +0000
@@ -385,6 +385,7 @@
385385
386class WebServiceConfiguration(BaseWebServiceConfiguration):386class WebServiceConfiguration(BaseWebServiceConfiguration):
387 directives.publication_class(WebServiceTestPublication)387 directives.publication_class(WebServiceTestPublication)
388 caching_policy = [10000, 2]
388 code_revision = 'test.revision'389 code_revision = 'test.revision'
389 default_batch_size = 5390 default_batch_size = 5
390 hostname = 'cookbooks.dev'391 hostname = 'cookbooks.dev'
391392
=== modified file 'src/lazr/restful/example/base/tests/root.txt'
--- src/lazr/restful/example/base/tests/root.txt 2010-02-11 17:57:16 +0000
+++ src/lazr/restful/example/base/tests/root.txt 2010-04-13 15:37:34 +0000
@@ -173,3 +173,91 @@
173173
174 >>> print top_level_links['featured_cookbook_link']174 >>> print top_level_links['featured_cookbook_link']
175 http://.../cookbooks/featured175 http://.../cookbooks/featured
176
177Caching policy
178==============
179
180The service root resource is served with the Cache-Control header
181giving a configurable value for "max-age". An old version of the
182service root can be cached for a long time:
183
184 >>> response = webservice.get('/', api_version='1.0')
185 >>> print response.getheader('Cache-Control')
186 max-age=10000
187
188The latest version of the service root should be cached for less time.
189
190 >>> response = webservice.get('/', api_version='devel')
191 >>> print response.getheader('Cache-Control')
192 max-age=2
193
194Both the WADL and JSON representations of the service root are
195cacheable.
196
197 >>> wadl_type = 'application/vnd.sun.wadl+xml'
198 >>> response = webservice.get('/', wadl_type)
199 >>> print response.getheader('Cache-Control')
200 max-age=2
201
202The Date header is set along with Cache-Control so that the client can
203easily determine when the cache is stale.
204
205 >>> response.getheader('Date') is None
206 False
207
208Date and Cache-Control are set even when the request is a conditional
209request where the condition failed.
210
211 >>> etag = response.getheader('ETag')
212 >>> conditional_response = webservice.get(
213 ... '/', wadl_type, headers={'If-None-Match' : etag})
214 >>> conditional_response.status
215 304
216 >>> print conditional_response.getheader('Cache-Control')
217 max-age=2
218 >>> conditional_response.getheader('Date') is None
219 False
220
221To avoid triggering a bug in httplib2, lazr.restful does not send the
222Cache-Control or Date headers to clients that identify as
223Python-httplib2.
224
225 # XXX leonardr 20100412
226 # bug=http://code.google.com/p/httplib2/issues/detail?id=97
227 >>> agent = 'Python-httplib2/$Rev: 259$'
228 >>> response = webservice.get(
229 ... '/', wadl_type, headers={'User-Agent' : agent})
230 >>> print response.getheader('Cache-Control')
231 None
232 >>> print response.getheader('Date')
233 None
234
235If the client identifies as an agent _based on_ httplib2, we take a
236chance and send the Cache-Control headers.
237
238 >>> agent = "Custom client (%s)" % agent
239 >>> response = webservice.get(
240 ... '/', wadl_type, headers={'User-Agent' : agent})
241 >>> print response.getheader('Cache-Control')
242 max-age=2
243 >>> response.getheader('Date') is None
244 False
245
246If the caching policy says not to cache the service root resource at
247all, the Cache-Control and Date headers are not present.
248
249 >>> from zope.component import getUtility
250 >>> from lazr.restful.interfaces import IWebServiceConfiguration
251 >>> policy = getUtility(IWebServiceConfiguration).caching_policy
252 >>> old_value = policy[-1]
253 >>> policy[-1] = 0
254
255 >>> response = webservice.get('/')
256 >>> print response.getheader('Cache-Control')
257 None
258 >>> response.getheader('Date') is None
259 True
260
261Cleanup.
262
263 >>> policy[-1] = old_value
176264
=== modified file 'src/lazr/restful/interfaces/_rest.py'
--- src/lazr/restful/interfaces/_rest.py 2010-03-10 19:03:11 +0000
+++ src/lazr/restful/interfaces/_rest.py 2010-04-13 15:37:34 +0000
@@ -63,6 +63,9 @@
63 IBrowserRequest, IDefaultBrowserLayer)63 IBrowserRequest, IDefaultBrowserLayer)
64from lazr.batchnavigator.interfaces import InvalidBatchSizeError64from lazr.batchnavigator.interfaces import InvalidBatchSizeError
6565
66# Constants for periods of time
67HOUR = 3600 # seconds
68
66# The namespace prefix for LAZR web service-related tags.69# The namespace prefix for LAZR web service-related tags.
67LAZR_WEBSERVICE_NS = 'lazr.restful'70LAZR_WEBSERVICE_NS = 'lazr.restful'
6871
@@ -394,6 +397,17 @@
394 These are miscellaneous strings that may differ in different web397 These are miscellaneous strings that may differ in different web
395 services.398 services.
396 """399 """
400 caching_policy = List(
401 value_type=Int(),
402 default = [7 * 24 * HOUR, 1 * HOUR],
403 title=u"The web service caching policy.",
404 description = u"""A list of two numbers, each to be used in the
405 'max-age' field of the Cache-Control header. The first number is
406 used when serving the service root for any web service version
407 except the latest one. The second number is used when serving the
408 service root for the latest version (which probably changes more
409 often).""")
410
397 service_description = TextLine(411 service_description = TextLine(
398 title=u"Service description",412 title=u"Service description",
399 description=u"""A human-readable description of the web service.413 description=u"""A human-readable description of the web service.

Subscribers

People subscribed via source and target branches