Launchpad itself

Merge lp:~mwhudson/launchpad/aiee-everything-is-broken into lp:launchpad

  1. aiee-everything-is-broken
  2. Merge into devel
Proposed by Michael Hudson-Doyle
Status: Rejected
Rejected by: Michael Hudson-Doyle
Proposed branch: lp:~mwhudson/launchpad/aiee-everything-is-broken
Merge into: lp:launchpad
Diff against target: 356 lines (+134/-86)
7 files modified
Makefile (+0/-4)
lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml (+0/-10)
lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt (+16/-0)
lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt (+68/-59)
lib/canonical/launchpad/systemhomes.py (+9/-4)
lib/canonical/launchpad/testing/pages.py (+34/-9)
lib/canonical/testing/layers.py (+7/-0)
To merge this branch: bzr merge lp:~mwhudson/launchpad/aiee-everything-is-broken
Reviewer Review Type Date Requested Status
Tim Penhey (community) Approve
Review via email: mp+22185@code.launchpad.net

Commit message

Fix implausible breakage in the test environment by changing the launchpadlib Launchpad object pagetest glob to a function that creates one. Fix a couple of tests that were failing but not detected as such.

Description of the change

http://<email address hidden>/msg02999.html contains the long winded explanation.

More fixes might have to be accumulated into this branch if the current run in ec2 finds more lurking test failures.

To post a comment you must log in.
lp:~mwhudson/launchpad/aiee-everything-is-broken updated
10598. By Michael Hudson-Doyle

fix xx-product-add.txt

10599. By Michael Hudson-Doyle

reenable the launchpadlib test using a new getLaunchpadObjectForPerson glob

10600. By Michael Hudson-Doyle

docstrings

Revision history for this message
Tim Penhey (thumper) wrote :

Looked at http://pastebin.ubuntu.com/401568/.

It's all good.

review: Approve
lp:~mwhudson/launchpad/aiee-everything-is-broken updated
10601. By Michael Hudson-Doyle

grr

10602. By Michael Hudson-Doyle

merge some trunk

10603. By Michael Hudson-Doyle

null merge trunk r10598.1.1

10604. By Michael Hudson-Doyle

merge rest of trunk

Unmerged revisions

10604. By Michael Hudson-Doyle

merge rest of trunk

10603. By Michael Hudson-Doyle

null merge trunk r10598.1.1

10602. By Michael Hudson-Doyle

merge some trunk

10601. By Michael Hudson-Doyle

grr

10600. By Michael Hudson-Doyle

docstrings

10599. By Michael Hudson-Doyle

reenable the launchpadlib test using a new getLaunchpadObjectForPerson glob

10598. By Michael Hudson-Doyle

fix xx-product-add.txt

10597. By William Grant

fix only known failure that has crept in

10596. By Michael Hudson-Doyle

remove launchpad from pagetest globs; bowlderize page test

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Makefile'
2--- Makefile 2010-03-26 13:23:20 +0000
3+++ Makefile 2010-03-28 20:04:28 +0000
4@@ -324,11 +324,7 @@
5 $(RM) -r lib/mailman
6 $(RM) -rf lib/canonical/launchpad/icing/build/*
7 $(RM) -r $(CODEHOSTING_ROOT)
8- mv $(APIDOC_DIR)/wadl-testrunner-devel.xml \
9- $(APIDOC_DIR)/wadl-testrunner-devel.xml.bak
10 $(RM) $(APIDOC_DIR)/wadl*.xml $(APIDOC_DIR)/*.html
11- mv $(APIDOC_DIR)/wadl-testrunner-devel.xml.bak \
12- $(APIDOC_DIR)/wadl-testrunner-devel.xml
13 $(RM) -rf $(APIDOC_DIR).tmp
14 $(RM) $(BZR_VERSION_INFO)
15 $(RM) _pythonpath.py
16
17=== removed file 'lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml'
18--- lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml 2010-03-26 16:47:42 +0000
19+++ lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml 1970-01-01 00:00:00 +0000
20@@ -1,10 +0,0 @@
21-<?xml version="1.0"?>
22-<wadl:application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
23- xmlns="http://research.sun.com/wadl/2006/10"
24- xmlns:wadl="http://research.sun.com/wadl/2006/10"
25- xsi:schemaLocation="http://research.sun.com/wadl/2006/10/wadl.xsd">
26-
27- <!--This file is for testing purposes only. See
28- canonical/launcpad/pagetests/webservice/xx-wadl.txt -->
29-
30-</wadl:application>
31
32=== added file 'lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt'
33--- lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt 1970-01-01 00:00:00 +0000
34+++ lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt 2010-03-28 20:04:28 +0000
35@@ -0,0 +1,16 @@
36+Using launchpadlib in pagetests
37+===============================
38+
39+As an alternative to crafting HTTP requests with the 'webservice'
40+object, you can write pagetests using launchpadlib. The
41+'getLaunchpadObjectForPerson' function returns a launchpadlib
42+Launchpad object that's pre-authenticated as the passed in
43+user. Anything you can do with 'webservice', you can do with this
44+Lanchpad object.
45+
46+ >>> login(ANONYMOUS)
47+ >>> bob = factory.makePerson(name='bob')
48+ >>> logout()
49+ >>> launchpad = getLaunchpadObjectForPerson(bob)
50+ >>> print launchpad.me.name
51+ bob
52
53=== modified file 'lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt'
54--- lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt 2010-03-26 13:23:20 +0000
55+++ lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt 2010-03-28 20:04:28 +0000
56@@ -3,55 +3,83 @@
57 Because Launchpad's main WADL files are so big, we cache them
58 internally: one WADL file for every version of the web service.
59 Because the WADL only changes when the Launchpad software changes,
60-these documents are cached to files. Right now, the Launchpad
61-webservice serves a special test file
62-(canonical/launchpad/apidoc/wadl-testrunner-devel.xml) when a client
63-asks for the big WADL definition for the 'devel' version. The
64-'testrunner' part comes from canonical.config.config.instance_name, so
65-a development instance will use the file
66-canonical/launchpad/apidoc/wadl-development-{version}.xml.
67-
68- >>> test_wadl = webservice.get(
69- ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body
70- >>> print test_wadl
71- <?xml version="1.0"?>
72- <wadl:application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
73- xmlns="http://research.sun.com/wadl/2006/10"
74- xmlns:wadl="http://research.sun.com/wadl/2006/10"
75- xsi:schemaLocation="http://research.sun.com/wadl/2006/10/wadl.xsd">
76- <BLANKLINE>
77- <!--This file is for testing purposes only. See
78- canonical/launcpad/pagetests/webservice/xx-wadl.txt -->
79- <BLANKLINE>
80- </wadl:application>
81- <BLANKLINE>
82-
83-Let's look at the real contents, though. To do this, we need to
84-deactivate the cache. Simply clearing it out will just cause it to be
85-filled up again.
86+these documents are cached to files.
87+
88+This test shows how the cache works. We'll start by temporarily
89+clearing the cache.
90
91 >>> from canonical.launchpad.systemhomes import WebServiceApplication
92 >>> old_cached_wadl = WebServiceApplication.cached_wadl
93- >>> WebServiceApplication.cached_wadl = None
94+ >>> WebServiceApplication.cached_wadl = {}
95+
96+If WADL is present in a certain file on disk--the filename depends on
97+the Launchpad configuration and the web service version--it will be
98+loaded from disk and not generated from scratch. But the testrunner
99+does not have any WADL files written to disk.
100+
101+ >>> import os
102+ >>> wadl_filename = WebServiceApplication.cachedWADLPath(
103+ ... 'testrunner', 'devel')
104+ >>> os.path.exists(wadl_filename)
105+ False
106+
107+Let's write some fake WADL to disk.
108+
109+ >>> fd = open(wadl_filename, "w")
110+ >>> fd.write("Some fake WADL.")
111+ >>> fd.close()
112+
113+When we request the WADL for version "devel", the fake WADL is loaded
114+from disk.
115+
116+ >>> print webservice.get(
117+ ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body
118+ Some fake WADL.
119+
120+The fake WADL is now present in the cache.
121+
122+ >>> WebServiceApplication.cached_wadl
123+ {u'devel': u'Some fake WADL.'}
124+
125+Change the cached value, and we change the document served.
126+
127+ >>> WebServiceApplication.cached_wadl['devel'] = "More fake WADL."
128+
129+ >>> print webservice.get(
130+ ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body
131+ More fake WADL.
132+
133+If there's no value in the cache and no cached file on disk, the WADL
134+is generated from scratch.
135+
136+ >>> WebServiceApplication.cached_wadl = {}
137+ >>> os.remove(wadl_filename)
138
139 >>> wadl = webservice.get(
140 ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body
141 >>> wadl = wadl.decode('UTF-8')
142
143-The real file is much bigger than the test file.
144-
145- >>> len(wadl) > len(test_wadl)
146+Unlike the test strings we used earlier, this is a valid WADL file.
147+
148+ >>> from lazr.restful import WADL_SCHEMA_FILE
149+ >>> from canonical.lazr.xml import XMLValidator
150+ >>> wadl_schema = XMLValidator(WADL_SCHEMA_FILE)
151+
152+ # We need to replace the nbsp entity, because the validator
153+ # doesn't support embedded definition.
154+ >>> wadl_schema.validate(
155+ ... wadl.replace('&nbsp;', '&#160;').encode('UTF-8'))
156 True
157
158 The WADL we received is keyed to the 'devel' version of the web
159-service. This version will always be present.
160+service. The URL to this version's service root will always be
161+present.
162
163 >>> 'http://api.launchpad.dev/devel/' in wadl
164 True
165
166 If we retrieve the WADL for the '1.0' version of the web service,
167-we'll get a document keyed to the '1.0' version. The '1.0' version
168-will be deprecated along with the Lucid release of Ubuntu.
169+we'll get a document keyed to the '1.0' version.
170
171 >>> wadl_10 = webservice.get(
172 ... '/', 'application/vd.sun.wadl+xml', api_version='1.0').body
173@@ -60,31 +88,12 @@
174 >>> 'http://api.launchpad.dev/1.0/' in wadl_10
175 True
176
177-If we retrieve the WADL for the 'beta' version of the web service,
178-we'll get a document keyed to the 'beta' version. The '1.0' version
179-will be deprecated along with the Karmic release of Ubuntu.
180-
181- >>> wadl_beta = webservice.get(
182- ... '/', 'application/vd.sun.wadl+xml', api_version='beta').body
183- >>> wadl_beta = wadl_beta.decode('UTF-8')
184-
185- >>> 'http://api.launchpad.dev/beta/' in wadl_beta
186- True
187-
188-We don't need the cache anymore, so we'll reinstate the testing
189-version. This way, other tests will have a clean slate.
190+All of these documents were cached as they were generated:
191+
192+ >>> sorted(WebServiceApplication.cached_wadl.keys())
193+ [u'1.0', u'devel']
194+
195+Finally, restore the cache so that other tests will have a clean
196+slate.
197
198 >>> WebServiceApplication.cached_wadl = old_cached_wadl
199-
200-Like all lazr.restful applications, Launchpad's web service generates
201-valid WADL.
202-
203- >>> from lazr.restful import WADL_SCHEMA_FILE
204- >>> from canonical.lazr.xml import XMLValidator
205- >>> wadl_schema = XMLValidator(WADL_SCHEMA_FILE)
206-
207- # We need to replace the nbsp entity, because the validator
208- # doesn't support embedded definition.
209- >>> wadl_schema.validate(
210- ... wadl.replace('&nbsp;', '&#160;').encode('UTF-8'))
211- True
212
213=== modified file 'lib/canonical/launchpad/systemhomes.py'
214--- lib/canonical/launchpad/systemhomes.py 2010-03-26 13:23:20 +0000
215+++ lib/canonical/launchpad/systemhomes.py 2010-03-28 20:04:28 +0000
216@@ -354,6 +354,13 @@
217
218 cached_wadl = {}
219
220+ @classmethod
221+ def cachedWADLPath(cls, instance_name, version):
222+ """Helper method to calculate the path to a cached WADL file."""
223+ return os.path.join(
224+ os.path.dirname(os.path.normpath(__file__)),
225+ 'apidoc', 'wadl-%s-%s.xml' % (instance_name, version))
226+
227 def toWADL(self):
228 """See `IWebServiceApplication`.
229
230@@ -370,10 +377,8 @@
231 return super(WebServiceApplication, self).toWADL()
232 if version not in self.__class__.cached_wadl:
233 # It's not cached. Look for it on disk.
234- _wadl_filename = os.path.join(
235- os.path.dirname(os.path.normpath(__file__)),
236- 'apidoc', 'wadl-%s-%s.xml' % (config.instance_name, version))
237-
238+ _wadl_filename = self.cachedWADLPath(
239+ config.instance_name, version)
240 _wadl_fd = None
241 try:
242 _wadl_fd = codecs.open(_wadl_filename, encoding='UTF-8')
243
244=== modified file 'lib/canonical/launchpad/testing/pages.py'
245--- lib/canonical/launchpad/testing/pages.py 2010-03-26 13:23:20 +0000
246+++ lib/canonical/launchpad/testing/pages.py 2010-03-28 20:04:28 +0000
247@@ -15,13 +15,16 @@
248 import re
249 import transaction
250 import unittest
251+from urlparse import urljoin
252
253 from BeautifulSoup import (
254 BeautifulSoup, CData, Comment, Declaration, NavigableString, PageElement,
255 ProcessingInstruction, SoupStrainer, Tag)
256+
257 from contrib.oauth import (
258 OAuthConsumer, OAuthRequest, OAuthSignatureMethod_PLAINTEXT, OAuthToken)
259-from urlparse import urljoin
260+
261+from launchpadlib.launchpad import Launchpad
262
263 from zope.app.testing.functional import HTTPCaller, SimpleCookie
264 from zope.component import getUtility
265@@ -647,14 +650,10 @@
266 return str(canonical_url(*args, **kwargs))
267
268
269-def webservice_for_person(person, consumer_key='launchpad-library',
270- permission=OAuthPermission.READ_PUBLIC,
271- context=None):
272- """Return a valid LaunchpadWebServiceCaller for the person.
273-
274- Use this method to create a way to test the webservice that doesn't depend
275- on sample data.
276- """
277+def _create_access_token_for_person(person, consumer_key,
278+ permission=OAuthPermission.READ_PUBLIC,
279+ context=None):
280+ """Create an IOAuthAccessToken for `person`."""
281 if person.is_team:
282 raise AssertionError('This cannot be used with teams.')
283 login(ANONYMOUS)
284@@ -666,9 +665,34 @@
285 request_token.review(person, permission, context)
286 access_token = request_token.createAccessToken()
287 logout()
288+ return access_token
289+
290+
291+def webservice_for_person(person, consumer_key='launchpad-library',
292+ permission=OAuthPermission.READ_PUBLIC,
293+ context=None):
294+ """Return a valid LaunchpadWebServiceCaller for the person.
295+
296+ Use this method to create a way to test the webservice that doesn't depend
297+ on sample data.
298+ """
299+ access_token = _create_access_token_for_person(
300+ person, consumer_key, permission, context)
301 return LaunchpadWebServiceCaller(consumer_key, access_token.key)
302
303
304+def getLaunchpadObjectForPerson(person, consumer_key='launchpad-library',
305+ permission=OAuthPermission.READ_PUBLIC,
306+ context=None):
307+ """Get a launchpadlib Launchpad object authenticated as the given person.
308+ """
309+ access_token = _create_access_token_for_person(
310+ person, consumer_key, permission, context)
311+ return Launchpad.login(
312+ consumer_key, access_token.key, access_token.secret,
313+ service_root='http://api.launchpad.dev/', version="devel")
314+
315+
316 def setupDTCBrowser():
317 """Testbrowser configured for Distribution Translations Coordinators.
318
319@@ -725,6 +749,7 @@
320 'launchpad-library', 'nopriv-read-nonprivate')
321 test.globs['anon_webservice'] = LaunchpadWebServiceCaller(
322 'launchpad-library', '')
323+ test.globs['getLaunchpadObjectForPerson'] = getLaunchpadObjectForPerson
324 test.globs['setupBrowser'] = setupBrowser
325 test.globs['setupDTCBrowser'] = setupDTCBrowser
326 test.globs['setupRosettaExpertBrowser'] = setupRosettaExpertBrowser
327
328=== modified file 'lib/canonical/testing/layers.py'
329--- lib/canonical/testing/layers.py 2010-03-26 13:23:20 +0000
330+++ lib/canonical/testing/layers.py 2010-03-28 20:04:28 +0000
331@@ -73,6 +73,7 @@
332 from storm.zope.interfaces import IZStorm
333 import transaction
334 import wsgi_intercept
335+from wsgi_intercept import httplib2_intercept
336
337 from lazr.restful.utils import safe_hasattr
338
339@@ -938,12 +939,18 @@
340 register_launchpad_request_publication_factories()
341 wsgi_intercept.add_wsgi_intercept(
342 'localhost', 80, lambda: wsgi_application)
343+ wsgi_intercept.add_wsgi_intercept(
344+ 'api.launchpad.dev', 80, lambda: wsgi_application)
345+ httplib2_intercept.install()
346+
347
348 @classmethod
349 @profiled
350 def tearDown(cls):
351 FunctionalLayer.isSetUp = False
352 wsgi_intercept.remove_wsgi_intercept('localhost', 80)
353+ wsgi_intercept.remove_wsgi_intercept('api.launchpad.dev', 80)
354+ httplib2_intercept.uninstall()
355 # Signal Layer cannot be torn down fully
356 raise NotImplementedError
357