Launchpad itself

Merge lp:~mwhudson/launchpad/aiee-everything-is-broken into lp:launchpad

  1. aiee-everything-is-broken
  2. Merge into devel
Proposed by Michael Hudson-Doyle
Status: Rejected
Rejected by: Michael Hudson-Doyle
Proposed branch: lp:~mwhudson/launchpad/aiee-everything-is-broken
Merge into: lp:launchpad
Diff against target: 356 lines (+134/-86)
7 files modified
Makefile (+0/-4)
lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml (+0/-10)
lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt (+16/-0)
lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt (+68/-59)
lib/canonical/launchpad/systemhomes.py (+9/-4)
lib/canonical/launchpad/testing/pages.py (+34/-9)
lib/canonical/testing/layers.py (+7/-0)
To merge this branch: bzr merge lp:~mwhudson/launchpad/aiee-everything-is-broken
Reviewer Review Type Date Requested Status
Tim Penhey (community) Approve
Review via email: mp+22185@code.launchpad.net

Commit message

Fix implausible breakage in the test environment by changing the launchpadlib Launchpad object pagetest glob to a function that creates one. Fix a couple of tests that were failing but not detected as such.

Description of the change

http://<email address hidden>/msg02999.html contains the long winded explanation.

More fixes might have to be accumulated into this branch if the current run in ec2 finds more lurking test failures.

To post a comment you must log in.
lp:~mwhudson/launchpad/aiee-everything-is-broken updated
10598. By Michael Hudson-Doyle

fix xx-product-add.txt

10599. By Michael Hudson-Doyle

reenable the launchpadlib test using a new getLaunchpadObjectForPerson glob

10600. By Michael Hudson-Doyle

docstrings

Revision history for this message
Tim Penhey (thumper) wrote :

Looked at http://pastebin.ubuntu.com/401568/.

It's all good.

review: Approve
lp:~mwhudson/launchpad/aiee-everything-is-broken updated
10601. By Michael Hudson-Doyle

grr

10602. By Michael Hudson-Doyle

merge some trunk

10603. By Michael Hudson-Doyle

null merge trunk r10598.1.1

10604. By Michael Hudson-Doyle

merge rest of trunk

Unmerged revisions

10604. By Michael Hudson-Doyle

merge rest of trunk

10603. By Michael Hudson-Doyle

null merge trunk r10598.1.1

10602. By Michael Hudson-Doyle

merge some trunk

10601. By Michael Hudson-Doyle

grr

10600. By Michael Hudson-Doyle

docstrings

10599. By Michael Hudson-Doyle

reenable the launchpadlib test using a new getLaunchpadObjectForPerson glob

10598. By Michael Hudson-Doyle

fix xx-product-add.txt

10597. By William Grant

fix only known failure that has crept in

10596. By Michael Hudson-Doyle

remove launchpad from pagetest globs; bowlderize page test

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'Makefile'
--- Makefile 2010-03-26 13:23:20 +0000
+++ Makefile 2010-03-28 20:04:28 +0000
@@ -324,11 +324,7 @@
324 $(RM) -r lib/mailman324 $(RM) -r lib/mailman
325 $(RM) -rf lib/canonical/launchpad/icing/build/*325 $(RM) -rf lib/canonical/launchpad/icing/build/*
326 $(RM) -r $(CODEHOSTING_ROOT)326 $(RM) -r $(CODEHOSTING_ROOT)
327 mv $(APIDOC_DIR)/wadl-testrunner-devel.xml \
328 $(APIDOC_DIR)/wadl-testrunner-devel.xml.bak
329 $(RM) $(APIDOC_DIR)/wadl*.xml $(APIDOC_DIR)/*.html327 $(RM) $(APIDOC_DIR)/wadl*.xml $(APIDOC_DIR)/*.html
330 mv $(APIDOC_DIR)/wadl-testrunner-devel.xml.bak \
331 $(APIDOC_DIR)/wadl-testrunner-devel.xml
332 $(RM) -rf $(APIDOC_DIR).tmp328 $(RM) -rf $(APIDOC_DIR).tmp
333 $(RM) $(BZR_VERSION_INFO)329 $(RM) $(BZR_VERSION_INFO)
334 $(RM) _pythonpath.py330 $(RM) _pythonpath.py
335331
=== removed file 'lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml'
--- lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml 2010-03-26 16:47:42 +0000
+++ lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml 1970-01-01 00:00:00 +0000
@@ -1,10 +0,0 @@
1<?xml version="1.0"?>
2<wadl:application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
3 xmlns="http://research.sun.com/wadl/2006/10"
4 xmlns:wadl="http://research.sun.com/wadl/2006/10"
5 xsi:schemaLocation="http://research.sun.com/wadl/2006/10/wadl.xsd">
6
7 <!--This file is for testing purposes only. See
8 canonical/launcpad/pagetests/webservice/xx-wadl.txt -->
9
10</wadl:application>
110
=== added file 'lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt'
--- lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt 1970-01-01 00:00:00 +0000
+++ lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt 2010-03-28 20:04:28 +0000
@@ -0,0 +1,16 @@
1Using launchpadlib in pagetests
2===============================
3
4As an alternative to crafting HTTP requests with the 'webservice'
5object, you can write pagetests using launchpadlib. The
6'getLaunchpadObjectForPerson' function returns a launchpadlib
7Launchpad object that's pre-authenticated as the passed in
8user. Anything you can do with 'webservice', you can do with this
9Lanchpad object.
10
11 >>> login(ANONYMOUS)
12 >>> bob = factory.makePerson(name='bob')
13 >>> logout()
14 >>> launchpad = getLaunchpadObjectForPerson(bob)
15 >>> print launchpad.me.name
16 bob
017
=== modified file 'lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt'
--- lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt 2010-03-26 13:23:20 +0000
+++ lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt 2010-03-28 20:04:28 +0000
@@ -3,55 +3,83 @@
3Because Launchpad's main WADL files are so big, we cache them3Because Launchpad's main WADL files are so big, we cache them
4internally: one WADL file for every version of the web service.4internally: one WADL file for every version of the web service.
5Because the WADL only changes when the Launchpad software changes,5Because the WADL only changes when the Launchpad software changes,
6these documents are cached to files. Right now, the Launchpad6these documents are cached to files.
7webservice serves a special test file7
8(canonical/launchpad/apidoc/wadl-testrunner-devel.xml) when a client8This test shows how the cache works. We'll start by temporarily
9asks for the big WADL definition for the 'devel' version. The9clearing the cache.
10'testrunner' part comes from canonical.config.config.instance_name, so
11a development instance will use the file
12canonical/launchpad/apidoc/wadl-development-{version}.xml.
13
14 >>> test_wadl = webservice.get(
15 ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body
16 >>> print test_wadl
17 <?xml version="1.0"?>
18 <wadl:application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
19 xmlns="http://research.sun.com/wadl/2006/10"
20 xmlns:wadl="http://research.sun.com/wadl/2006/10"
21 xsi:schemaLocation="http://research.sun.com/wadl/2006/10/wadl.xsd">
22 <BLANKLINE>
23 <!--This file is for testing purposes only. See
24 canonical/launcpad/pagetests/webservice/xx-wadl.txt -->
25 <BLANKLINE>
26 </wadl:application>
27 <BLANKLINE>
28
29Let's look at the real contents, though. To do this, we need to
30deactivate the cache. Simply clearing it out will just cause it to be
31filled up again.
3210
33 >>> from canonical.launchpad.systemhomes import WebServiceApplication11 >>> from canonical.launchpad.systemhomes import WebServiceApplication
34 >>> old_cached_wadl = WebServiceApplication.cached_wadl12 >>> old_cached_wadl = WebServiceApplication.cached_wadl
35 >>> WebServiceApplication.cached_wadl = None13 >>> WebServiceApplication.cached_wadl = {}
14
15If WADL is present in a certain file on disk--the filename depends on
16the Launchpad configuration and the web service version--it will be
17loaded from disk and not generated from scratch. But the testrunner
18does not have any WADL files written to disk.
19
20 >>> import os
21 >>> wadl_filename = WebServiceApplication.cachedWADLPath(
22 ... 'testrunner', 'devel')
23 >>> os.path.exists(wadl_filename)
24 False
25
26Let's write some fake WADL to disk.
27
28 >>> fd = open(wadl_filename, "w")
29 >>> fd.write("Some fake WADL.")
30 >>> fd.close()
31
32When we request the WADL for version "devel", the fake WADL is loaded
33from disk.
34
35 >>> print webservice.get(
36 ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body
37 Some fake WADL.
38
39The fake WADL is now present in the cache.
40
41 >>> WebServiceApplication.cached_wadl
42 {u'devel': u'Some fake WADL.'}
43
44Change the cached value, and we change the document served.
45
46 >>> WebServiceApplication.cached_wadl['devel'] = "More fake WADL."
47
48 >>> print webservice.get(
49 ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body
50 More fake WADL.
51
52If there's no value in the cache and no cached file on disk, the WADL
53is generated from scratch.
54
55 >>> WebServiceApplication.cached_wadl = {}
56 >>> os.remove(wadl_filename)
3657
37 >>> wadl = webservice.get(58 >>> wadl = webservice.get(
38 ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body59 ... '/', 'application/vd.sun.wadl+xml', api_version='devel').body
39 >>> wadl = wadl.decode('UTF-8')60 >>> wadl = wadl.decode('UTF-8')
4061
41The real file is much bigger than the test file.62Unlike the test strings we used earlier, this is a valid WADL file.
4263
43 >>> len(wadl) > len(test_wadl)64 >>> from lazr.restful import WADL_SCHEMA_FILE
65 >>> from canonical.lazr.xml import XMLValidator
66 >>> wadl_schema = XMLValidator(WADL_SCHEMA_FILE)
67
68 # We need to replace the nbsp entity, because the validator
69 # doesn't support embedded definition.
70 >>> wadl_schema.validate(
71 ... wadl.replace('&nbsp;', '&#160;').encode('UTF-8'))
44 True72 True
4573
46The WADL we received is keyed to the 'devel' version of the web74The WADL we received is keyed to the 'devel' version of the web
47service. This version will always be present.75service. The URL to this version's service root will always be
76present.
4877
49 >>> 'http://api.launchpad.dev/devel/' in wadl78 >>> 'http://api.launchpad.dev/devel/' in wadl
50 True79 True
5180
52If we retrieve the WADL for the '1.0' version of the web service,81If we retrieve the WADL for the '1.0' version of the web service,
53we'll get a document keyed to the '1.0' version. The '1.0' version82we'll get a document keyed to the '1.0' version.
54will be deprecated along with the Lucid release of Ubuntu.
5583
56 >>> wadl_10 = webservice.get(84 >>> wadl_10 = webservice.get(
57 ... '/', 'application/vd.sun.wadl+xml', api_version='1.0').body85 ... '/', 'application/vd.sun.wadl+xml', api_version='1.0').body
@@ -60,31 +88,12 @@
60 >>> 'http://api.launchpad.dev/1.0/' in wadl_1088 >>> 'http://api.launchpad.dev/1.0/' in wadl_10
61 True89 True
6290
63If we retrieve the WADL for the 'beta' version of the web service,91All of these documents were cached as they were generated:
64we'll get a document keyed to the 'beta' version. The '1.0' version92
65will be deprecated along with the Karmic release of Ubuntu.93 >>> sorted(WebServiceApplication.cached_wadl.keys())
6694 [u'1.0', u'devel']
67 >>> wadl_beta = webservice.get(95
68 ... '/', 'application/vd.sun.wadl+xml', api_version='beta').body96Finally, restore the cache so that other tests will have a clean
69 >>> wadl_beta = wadl_beta.decode('UTF-8')97slate.
70
71 >>> 'http://api.launchpad.dev/beta/' in wadl_beta
72 True
73
74We don't need the cache anymore, so we'll reinstate the testing
75version. This way, other tests will have a clean slate.
7698
77 >>> WebServiceApplication.cached_wadl = old_cached_wadl99 >>> WebServiceApplication.cached_wadl = old_cached_wadl
78
79Like all lazr.restful applications, Launchpad's web service generates
80valid WADL.
81
82 >>> from lazr.restful import WADL_SCHEMA_FILE
83 >>> from canonical.lazr.xml import XMLValidator
84 >>> wadl_schema = XMLValidator(WADL_SCHEMA_FILE)
85
86 # We need to replace the nbsp entity, because the validator
87 # doesn't support embedded definition.
88 >>> wadl_schema.validate(
89 ... wadl.replace('&nbsp;', '&#160;').encode('UTF-8'))
90 True
91100
=== modified file 'lib/canonical/launchpad/systemhomes.py'
--- lib/canonical/launchpad/systemhomes.py 2010-03-26 13:23:20 +0000
+++ lib/canonical/launchpad/systemhomes.py 2010-03-28 20:04:28 +0000
@@ -354,6 +354,13 @@
354354
355 cached_wadl = {}355 cached_wadl = {}
356356
357 @classmethod
358 def cachedWADLPath(cls, instance_name, version):
359 """Helper method to calculate the path to a cached WADL file."""
360 return os.path.join(
361 os.path.dirname(os.path.normpath(__file__)),
362 'apidoc', 'wadl-%s-%s.xml' % (instance_name, version))
363
357 def toWADL(self):364 def toWADL(self):
358 """See `IWebServiceApplication`.365 """See `IWebServiceApplication`.
359366
@@ -370,10 +377,8 @@
370 return super(WebServiceApplication, self).toWADL()377 return super(WebServiceApplication, self).toWADL()
371 if version not in self.__class__.cached_wadl:378 if version not in self.__class__.cached_wadl:
372 # It's not cached. Look for it on disk.379 # It's not cached. Look for it on disk.
373 _wadl_filename = os.path.join(380 _wadl_filename = self.cachedWADLPath(
374 os.path.dirname(os.path.normpath(__file__)),381 config.instance_name, version)
375 'apidoc', 'wadl-%s-%s.xml' % (config.instance_name, version))
376
377 _wadl_fd = None382 _wadl_fd = None
378 try:383 try:
379 _wadl_fd = codecs.open(_wadl_filename, encoding='UTF-8')384 _wadl_fd = codecs.open(_wadl_filename, encoding='UTF-8')
380385
=== modified file 'lib/canonical/launchpad/testing/pages.py'
--- lib/canonical/launchpad/testing/pages.py 2010-03-26 13:23:20 +0000
+++ lib/canonical/launchpad/testing/pages.py 2010-03-28 20:04:28 +0000
@@ -15,13 +15,16 @@
15import re15import re
16import transaction16import transaction
17import unittest17import unittest
18from urlparse import urljoin
1819
19from BeautifulSoup import (20from BeautifulSoup import (
20 BeautifulSoup, CData, Comment, Declaration, NavigableString, PageElement,21 BeautifulSoup, CData, Comment, Declaration, NavigableString, PageElement,
21 ProcessingInstruction, SoupStrainer, Tag)22 ProcessingInstruction, SoupStrainer, Tag)
23
22from contrib.oauth import (24from contrib.oauth import (
23 OAuthConsumer, OAuthRequest, OAuthSignatureMethod_PLAINTEXT, OAuthToken)25 OAuthConsumer, OAuthRequest, OAuthSignatureMethod_PLAINTEXT, OAuthToken)
24from urlparse import urljoin26
27from launchpadlib.launchpad import Launchpad
2528
26from zope.app.testing.functional import HTTPCaller, SimpleCookie29from zope.app.testing.functional import HTTPCaller, SimpleCookie
27from zope.component import getUtility30from zope.component import getUtility
@@ -647,14 +650,10 @@
647 return str(canonical_url(*args, **kwargs))650 return str(canonical_url(*args, **kwargs))
648651
649652
650def webservice_for_person(person, consumer_key='launchpad-library',653def _create_access_token_for_person(person, consumer_key,
651 permission=OAuthPermission.READ_PUBLIC,654 permission=OAuthPermission.READ_PUBLIC,
652 context=None):655 context=None):
653 """Return a valid LaunchpadWebServiceCaller for the person.656 """Create an IOAuthAccessToken for `person`."""
654
655 Use this method to create a way to test the webservice that doesn't depend
656 on sample data.
657 """
658 if person.is_team:657 if person.is_team:
659 raise AssertionError('This cannot be used with teams.')658 raise AssertionError('This cannot be used with teams.')
660 login(ANONYMOUS)659 login(ANONYMOUS)
@@ -666,9 +665,34 @@
666 request_token.review(person, permission, context)665 request_token.review(person, permission, context)
667 access_token = request_token.createAccessToken()666 access_token = request_token.createAccessToken()
668 logout()667 logout()
668 return access_token
669
670
671def webservice_for_person(person, consumer_key='launchpad-library',
672 permission=OAuthPermission.READ_PUBLIC,
673 context=None):
674 """Return a valid LaunchpadWebServiceCaller for the person.
675
676 Use this method to create a way to test the webservice that doesn't depend
677 on sample data.
678 """
679 access_token = _create_access_token_for_person(
680 person, consumer_key, permission, context)
669 return LaunchpadWebServiceCaller(consumer_key, access_token.key)681 return LaunchpadWebServiceCaller(consumer_key, access_token.key)
670682
671683
684def getLaunchpadObjectForPerson(person, consumer_key='launchpad-library',
685 permission=OAuthPermission.READ_PUBLIC,
686 context=None):
687 """Get a launchpadlib Launchpad object authenticated as the given person.
688 """
689 access_token = _create_access_token_for_person(
690 person, consumer_key, permission, context)
691 return Launchpad.login(
692 consumer_key, access_token.key, access_token.secret,
693 service_root='http://api.launchpad.dev/', version="devel")
694
695
672def setupDTCBrowser():696def setupDTCBrowser():
673 """Testbrowser configured for Distribution Translations Coordinators.697 """Testbrowser configured for Distribution Translations Coordinators.
674698
@@ -725,6 +749,7 @@
725 'launchpad-library', 'nopriv-read-nonprivate')749 'launchpad-library', 'nopriv-read-nonprivate')
726 test.globs['anon_webservice'] = LaunchpadWebServiceCaller(750 test.globs['anon_webservice'] = LaunchpadWebServiceCaller(
727 'launchpad-library', '')751 'launchpad-library', '')
752 test.globs['getLaunchpadObjectForPerson'] = getLaunchpadObjectForPerson
728 test.globs['setupBrowser'] = setupBrowser753 test.globs['setupBrowser'] = setupBrowser
729 test.globs['setupDTCBrowser'] = setupDTCBrowser754 test.globs['setupDTCBrowser'] = setupDTCBrowser
730 test.globs['setupRosettaExpertBrowser'] = setupRosettaExpertBrowser755 test.globs['setupRosettaExpertBrowser'] = setupRosettaExpertBrowser
731756
=== modified file 'lib/canonical/testing/layers.py'
--- lib/canonical/testing/layers.py 2010-03-26 13:23:20 +0000
+++ lib/canonical/testing/layers.py 2010-03-28 20:04:28 +0000
@@ -73,6 +73,7 @@
73from storm.zope.interfaces import IZStorm73from storm.zope.interfaces import IZStorm
74import transaction74import transaction
75import wsgi_intercept75import wsgi_intercept
76from wsgi_intercept import httplib2_intercept
7677
77from lazr.restful.utils import safe_hasattr78from lazr.restful.utils import safe_hasattr
7879
@@ -938,12 +939,18 @@
938 register_launchpad_request_publication_factories()939 register_launchpad_request_publication_factories()
939 wsgi_intercept.add_wsgi_intercept(940 wsgi_intercept.add_wsgi_intercept(
940 'localhost', 80, lambda: wsgi_application)941 'localhost', 80, lambda: wsgi_application)
942 wsgi_intercept.add_wsgi_intercept(
943 'api.launchpad.dev', 80, lambda: wsgi_application)
944 httplib2_intercept.install()
945
941946
942 @classmethod947 @classmethod
943 @profiled948 @profiled
944 def tearDown(cls):949 def tearDown(cls):
945 FunctionalLayer.isSetUp = False950 FunctionalLayer.isSetUp = False
946 wsgi_intercept.remove_wsgi_intercept('localhost', 80)951 wsgi_intercept.remove_wsgi_intercept('localhost', 80)
952 wsgi_intercept.remove_wsgi_intercept('api.launchpad.dev', 80)
953 httplib2_intercept.uninstall()
947 # Signal Layer cannot be torn down fully954 # Signal Layer cannot be torn down fully
948 raise NotImplementedError955 raise NotImplementedError
949956