lazr.restful

Merge lp:~leonardr/lazr.restful/integrate-optimized-len into lp:lazr.restful

  1. integrate-optimized-len
  2. Merge into trunk
Proposed by Leonard Richardson
Status: Merged
Approved by: Edwin Grubbs
Approved revision: 142
Merged at revision: 140
Proposed branch: lp:~leonardr/lazr.restful/integrate-optimized-len
Merge into: lp:lazr.restful
Diff against target: 283 lines (+90/-20)
9 files modified
src/lazr/restful/NEWS.txt (+7/-0)
src/lazr/restful/_resource.py (+13/-4)
src/lazr/restful/docs/multiversion.txt (+10/-5)
src/lazr/restful/example/base/tests/collection.txt (+28/-3)
src/lazr/restful/example/multiversion/root.py (+2/-0)
src/lazr/restful/example/multiversion/tests/introduction.txt (+12/-0)
src/lazr/restful/example/multiversion/tests/operation.txt (+16/-6)
src/lazr/restful/version.txt (+1/-1)
versions.cfg (+1/-1)
To merge this branch: bzr merge lp:~leonardr/lazr.restful/integrate-optimized-len
Reviewer Review Type Date Requested Status
Edwin Grubbs (community) code Approve
Review via email: mp+33023@code.launchpad.net

Description of the change

Thanks to an optimization in lazr.batchnavigator, it's now very cheap to determine the total size of a list if the current batch is the last one in the list. It's also cheap to determine whether the current batch is the last one or not. (The most interesting case for our purposes is when the entire list fits in one batch, in which case the first batch is also the last.)

This branch backs lazr.restful away a little bit from its absolute insistence on sending 'total_size_link' instead of 'total_size' in responses to named operations. If the total size of the list turns out to be cheap to calculate, lazr.restful calculates it and sends 'total_size' instead of 'total_size_link'.

Most of this branch consists of test changes: creating scenarios in which the optimization doesn't apply, and showing that total_size_link is sent; creating scenarios in which the optimization does apply, and showing that total_size is sent instead. In one case this required creating extra sample data to pad out a collection returned from a named operation.

To post a comment you must log in.
lp:~leonardr/lazr.restful/integrate-optimized-len updated
141. By Leonard Richardson

updated NEWS.

142. By Leonard Richardson

Use total(), which is part of the public interface, instead of .listlength.

Revision history for this message
Edwin Grubbs (edwin-grubbs) wrote :

Hi Leonard,

This looks good.

-Edwin

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'src/lazr/restful/NEWS.txt'
--- src/lazr/restful/NEWS.txt 2010-08-13 14:31:39 +0000
+++ src/lazr/restful/NEWS.txt 2010-08-18 16:44:44 +0000
@@ -2,6 +2,13 @@
2NEWS for lazr.restful2NEWS for lazr.restful
3=====================3=====================
44
50.11.2 (Unreleased)
6===================
7
8Optimized lazr.restful to send 'total_size' instead of
9'total_size_link' when 'total_size' is easy to calculate, possibly
10saving the client from sending another HTTP request.
11
50.11.1 (2010-08-13)120.11.1 (2010-08-13)
6===================13===================
714
815
=== modified file 'src/lazr/restful/_resource.py'
--- src/lazr/restful/_resource.py 2010-08-09 20:36:57 +0000
+++ src/lazr/restful/_resource.py 2010-08-18 16:44:44 +0000
@@ -640,13 +640,22 @@
640640
641 view_permission = getUtility(IWebServiceConfiguration).view_permission641 view_permission = getUtility(IWebServiceConfiguration).view_permission
642 batch = { 'start' : navigator.batch.start }642 batch = { 'start' : navigator.batch.start }
643 total_size_link = self.total_size_link(navigator)643 # If this is the last batch, then total() is easy to
644 # calculate. Let's use it and save the client from having to
645 # ask for the total size of the collection.
646 if navigator.batch.nextBatch() is None:
647 total_size_link = None
648 else:
649 total_size_link = self.total_size_link(navigator)
644 if total_size_link is None:650 if total_size_link is None:
645 batch['total_size'] = navigator.batch.listlength651 total_size = navigator.batch.total()
652 if total_size < 0:
653 # lazr.batchnavigator uses -1 for an empty list.
654 # We want to use 0.
655 total_size = 0
656 batch['total_size'] = total_size
646 else:657 else:
647 batch['total_size_link'] = total_size_link658 batch['total_size_link'] = total_size_link
648 if navigator.batch.start < 0:
649 batch['start'] = None
650 next_url = navigator.nextBatchURL()659 next_url = navigator.nextBatchURL()
651 if next_url != "":660 if next_url != "":
652 batch['next_collection_link'] = next_url661 batch['next_collection_link'] = next_url
653662
=== modified file 'src/lazr/restful/docs/multiversion.txt'
--- src/lazr/restful/docs/multiversion.txt 2010-08-13 14:29:52 +0000
+++ src/lazr/restful/docs/multiversion.txt 2010-08-18 16:44:44 +0000
@@ -857,11 +857,11 @@
857857
858 >>> request_10 = create_web_service_request(858 >>> request_10 = create_web_service_request(
859 ... '/1.0/contacts',859 ... '/1.0/contacts',
860 ... environ={'QUERY_STRING' : 'ws.op=find&string=Cleo'})860 ... environ={'QUERY_STRING' : 'ws.op=find&string=e&ws.size=2'})
861 >>> operation = request_10.traverse(None)861 >>> operation = request_10.traverse(None)
862 >>> result = simplejson.loads(operation())862 >>> result = simplejson.loads(operation())
863 >>> [contact['name'] for contact in result['entries']]863 >>> [contact['name'] for contact in result['entries']]
864 ['Cleo Python']864 ['Cleo Python', 'Oliver Bluth']
865865
866 >>> result['total_size']866 >>> result['total_size']
867 Traceback (most recent call last):867 Traceback (most recent call last):
@@ -869,15 +869,18 @@
869 KeyError: 'total_size'869 KeyError: 'total_size'
870870
871 >>> print result['total_size_link']871 >>> print result['total_size_link']
872 http://.../1.0/contacts?string=Cleo&ws.op=find&ws.show=total_size872 http://.../1.0/contacts?string=e&ws.op=find&ws.show=total_size
873 >>> size_request = create_web_service_request(873 >>> size_request = create_web_service_request(
874 ... '/1.0/contacts',874 ... '/1.0/contacts',
875 ... environ={'QUERY_STRING' :875 ... environ={'QUERY_STRING' :
876 ... 'string=Cleo&ws.op=find&ws.show=total_size'})876 ... 'string=e&ws.op=find&ws.show=total_size'})
877 >>> operation = size_request.traverse(None)877 >>> operation = size_request.traverse(None)
878 >>> result = simplejson.loads(operation())878 >>> result = simplejson.loads(operation())
879 >>> print result879 >>> print result
880 1880 3
881
882If the resultset fits on a single page, total_size will be provided
883instead of total_size_link, as a convenience.
881884
882 >>> request_10 = create_web_service_request(885 >>> request_10 = create_web_service_request(
883 ... '/1.0/contacts',886 ... '/1.0/contacts',
@@ -886,6 +889,8 @@
886 >>> result = simplejson.loads(operation())889 >>> result = simplejson.loads(operation())
887 >>> [contact['fax_number'] for contact in result['entries']]890 >>> [contact['fax_number'] for contact in result['entries']]
888 ['111-2121']891 ['111-2121']
892 >>> result['total_size']
893 1
889894
890Attempting to invoke the operation using its 'beta' name won't work.895Attempting to invoke the operation using its 'beta' name won't work.
891896
892897
=== modified file 'src/lazr/restful/example/base/tests/collection.txt'
--- src/lazr/restful/example/base/tests/collection.txt 2010-08-10 11:38:06 +0000
+++ src/lazr/restful/example/base/tests/collection.txt 2010-08-18 16:44:44 +0000
@@ -209,6 +209,32 @@
209 >>> print webservice.get(s_recipes['total_size_link']).jsonBody()209 >>> print webservice.get(s_recipes['total_size_link']).jsonBody()
210 3210 3
211211
212If the entire collection fits in a single 'page' of results, the
213'total_size_link' is not present; instead, lazr.restful provides the
214total size as a convenience to the client.
215
216 >>> full_list = search_recipes("chicken", size=100)
217 >>> len(full_list['entries'])
218 3
219 >>> full_list['total_size']
220 3
221 >>> full_list['total_size_link']
222 Traceback (most recent call last):
223 ...
224 KeyError: 'total_size_link'
225
226The same is true if the client requests the last page of a list.
227
228 >>> last_page = search_recipes("chicken", start=2, size=2)
229 >>> len(last_page['entries'])
230 1
231 >>> full_list['total_size']
232 3
233 >>> full_list['total_size_link']
234 Traceback (most recent call last):
235 ...
236 KeyError: 'total_size_link'
237
212Custom operations may have error handling. In this case, the error238Custom operations may have error handling. In this case, the error
213handling is in the validate() method of the 'search' field.239handling is in the validate() method of the 'search' field.
214240
@@ -224,9 +250,8 @@
224250
225 >>> general_cookbooks = webservice.get(251 >>> general_cookbooks = webservice.get(
226 ... "/cookbooks?ws.op=find_for_cuisine&cuisine=General")252 ... "/cookbooks?ws.op=find_for_cuisine&cuisine=General")
227 >>> print general_cookbooks.jsonBody()['total_size_link']253 >>> print general_cookbooks.jsonBody()['total_size']
228 http://cookbooks.dev/devel/cookbooks?cuisine=General&ws.op=find_for_cuisine&ws.show=total_size254 3
229
230255
231POST operations256POST operations
232===============257===============
233258
=== modified file 'src/lazr/restful/example/multiversion/root.py'
--- src/lazr/restful/example/multiversion/root.py 2010-08-13 14:29:52 +0000
+++ src/lazr/restful/example/multiversion/root.py 2010-08-18 16:44:44 +0000
@@ -47,6 +47,8 @@
47 pairset = PairSet()47 pairset = PairSet()
48 pairset.pairs = [48 pairset.pairs = [
49 KeyValuePair(pairset, "foo", "bar"),49 KeyValuePair(pairset, "foo", "bar"),
50 KeyValuePair(pairset, "foo2", "bar"),
51 KeyValuePair(pairset, "foo3", "bar"),
50 KeyValuePair(pairset, "1", "2"),52 KeyValuePair(pairset, "1", "2"),
51 KeyValuePair(pairset, "Some", None),53 KeyValuePair(pairset, "Some", None),
52 KeyValuePair(pairset, "Delete", "me"),54 KeyValuePair(pairset, "Delete", "me"),
5355
=== modified file 'src/lazr/restful/example/multiversion/tests/introduction.txt'
--- src/lazr/restful/example/multiversion/tests/introduction.txt 2010-08-13 14:29:52 +0000
+++ src/lazr/restful/example/multiversion/tests/introduction.txt 2010-08-18 16:44:44 +0000
@@ -86,12 +86,16 @@
86 Also delete: me86 Also delete: me
87 Delete: me87 Delete: me
88 foo: bar88 foo: bar
89 foo2: bar
90 foo3: bar
8991
90 >>> show_pairs('1.0')92 >>> show_pairs('1.0')
91 1: 293 1: 2
92 Also delete: me94 Also delete: me
93 Delete: me95 Delete: me
94 foo: bar96 foo: bar
97 foo2: bar
98 foo3: bar
9599
96 >>> show_pairs('2.0')100 >>> show_pairs('2.0')
97 1: 2101 1: 2
@@ -99,6 +103,8 @@
99 Delete: me103 Delete: me
100 Some: None104 Some: None
101 foo: bar105 foo: bar
106 foo2: bar
107 foo3: bar
102108
103 >>> show_pairs('3.0')109 >>> show_pairs('3.0')
104 1: 2110 1: 2
@@ -106,6 +112,8 @@
106 Delete: me112 Delete: me
107 Some: None113 Some: None
108 foo: bar114 foo: bar
115 foo2: bar
116 foo3: bar
109117
110Entries118Entries
111=======119=======
@@ -199,6 +207,8 @@
199 1: 2207 1: 2
200 Also delete: me208 Also delete: me
201 foo: bar209 foo: bar
210 foo2: bar
211 foo3: bar
202212
203In '3.0', deleting a key-value pair simply sets its 'deleted' field213In '3.0', deleting a key-value pair simply sets its 'deleted' field
204to True. (This is an abuse of the HTTP DELETE method, but it makes a214to True. (This is an abuse of the HTTP DELETE method, but it makes a
@@ -217,6 +227,8 @@
217 1: 2227 1: 2
218 Also delete: me228 Also delete: me
219 foo: bar229 foo: bar
230 foo2: bar
231 foo3: bar
220232
221And in a version which publishes the 'delete' field, we can check the233And in a version which publishes the 'delete' field, we can check the
222key-value pair's value for that field.234key-value pair's value for that field.
223235
=== modified file 'src/lazr/restful/example/multiversion/tests/operation.txt'
--- src/lazr/restful/example/multiversion/tests/operation.txt 2010-08-13 14:29:52 +0000
+++ src/lazr/restful/example/multiversion/tests/operation.txt 2010-08-18 16:44:44 +0000
@@ -24,26 +24,36 @@
24 >>> print config.first_version_with_total_size_link24 >>> print config.first_version_with_total_size_link
25 2.025 2.0
2626
27When the 'byValue' operation is invoked in version 1.0, it returns a27When the 'byValue' operation is invoked in version 1.0, it always returns a
28total_size.28total_size.
2929
30 >>> def get_collection(version, op='byValue'):30 >>> def get_collection(version, op='byValue', value="bar", size=2):
31 ... url = '/pairs?ws.op=%s&value=bar' % op31 ... url = '/pairs?ws.op=%s&value=%s&ws.size=%s' % (op, value, size)
32 ... return webservice.get(url, api_version=version).jsonBody()32 ... return webservice.get(url, api_version=version).jsonBody()
3333
34 >>> print sorted(get_collection('1.0').keys())34 >>> print sorted(get_collection('1.0').keys())
35 [u'entries', u'start', u'total_size']35 [u'entries', u'next_collection_link', u'start', u'total_size']
3636
37The operation itself doesn't change between 1.0 and 2.0, but in37The operation itself doesn't change between 1.0 and 2.0, but in
38version 2.0, the operation starts returning total_size_link.38version 2.0, the operation starts returning total_size_link.
3939
40 >>> print sorted(get_collection('2.0').keys())40 >>> print sorted(get_collection('2.0').keys())
41 [u'entries', u'start', u'total_size_link']41 [u'entries', u'next_collection_link', u'start', u'total_size_link']
4242
43The same happens in 3.0.43The same happens in 3.0.
4444
45 >>> print sorted(get_collection('3.0', 'by_value').keys())45 >>> print sorted(get_collection('3.0', 'by_value').keys())
46 [u'entries', u'start', u'total_size_link']46 [u'entries', u'next_collection_link', u'start', u'total_size_link']
47
48However, if the total size is easy to calculate (for instance, because
49all the results fit on one page), a total_size is returned instead of
50total_size_link.
51
52 >>> print sorted(get_collection('3.0', 'by_value', size=100).keys())
53 [u'entries', u'start', u'total_size']
54
55 >>> print get_collection('3.0', 'by_value', 'no-such-value')
56 {u'total_size': 0, u'start': 0, u'entries': []}
4757
48Mutators as named operations58Mutators as named operations
49----------------------------59----------------------------
5060
=== modified file 'src/lazr/restful/version.txt'
--- src/lazr/restful/version.txt 2010-08-13 14:31:39 +0000
+++ src/lazr/restful/version.txt 2010-08-18 16:44:44 +0000
@@ -1,1 +1,1 @@
10.11.110.11.2
22
=== modified file 'versions.cfg'
--- versions.cfg 2010-08-10 18:36:09 +0000
+++ versions.cfg 2010-08-18 16:44:44 +0000
@@ -16,7 +16,7 @@
16docutils = 0.516docutils = 0.5
17epydoc = 3.0.117epydoc = 3.0.1
18grokcore.component = 1.618grokcore.component = 1.6
19lazr.batchnavigator = 1.2.019lazr.batchnavigator = 1.2.2
20lazr.delegates = 1.2.020lazr.delegates = 1.2.0
21lazr.enum = 1.1.221lazr.enum = 1.1.2
22lazr.lifecycle = 1.022lazr.lifecycle = 1.0

Subscribers

People subscribed via source and target branches