Launchpad itself

Merge lp:~lifeless/launchpad/doctest into lp:launchpad

  1. doctest
  2. Merge into devel
Proposed by Robert Collins
Status: Merged
Approved by: Jonathan Lange
Approved revision: no longer in the source branch.
Merged at revision: 11723
Proposed branch: lp:~lifeless/launchpad/doctest
Merge into: lp:launchpad
Diff against target: 302 lines (+28/-80)
15 files modified
lib/canonical/launchpad/mail/ftests/test_stub.py (+0/-4)
lib/canonical/launchpad/mail/tests/test_commands.py (+1/-8)
lib/canonical/launchpad/mail/tests/test_handlers.py (+0/-4)
lib/canonical/launchpad/mail/tests/test_helpers.py (+1/-2)
lib/canonical/launchpad/scripts/tests/test_rundoctests.py (+0/-4)
lib/canonical/launchpad/testing/pages.py (+20/-13)
lib/canonical/launchpad/testing/tests/test_pages.py (+3/-1)
lib/canonical/launchpad/tests/test_datetimeutils.py (+1/-8)
lib/canonical/launchpad/tests/test_helpers.py (+0/-4)
lib/canonical/launchpad/webapp/tests/test_publisher.py (+0/-5)
lib/canonical/testing/tests/test_doctestcodec.py (+1/-5)
lib/canonical/tests/test_base.py (+0/-6)
lib/canonical/tests/test_encoding.py (+0/-5)
lib/canonical/widgets/ftests/test_widget_doctests.py (+0/-4)
lib/lp/services/mail/tests/test_incoming.py (+1/-7)
To merge this branch: bzr merge lp:~lifeless/launchpad/doctest
Reviewer Review Type Date Requested Status
Jonathan Lange (community) Approve
Review via email: mp+38182@code.launchpad.net

Commit message

Change the reported test id for all elements of a story to be the id of the story.

Description of the change

Prep work for parallel testing. One of the main components of parallel testing is programmatic rearrangement of the tests to run, and story tests in particular were a problem.

Previously the test *id* for the whole story was pagetest: relname. This is ok, but I've changed it to story- to make it more clear that this is for *stories*, not just pagetests.

And now the problem bit: the delegated tests were reporting their own ids; those ids don't exist (naturally, the story appearing as a single test so it can't be split hides the components) - and even if they did exist in the full test list, they can't be run separately.

So this branch changes the ids of the individual doctests so that they all claim to be the story as far as subunit and testr is concerned. This makes the full test list able to be partitioned and parallel tested without skipping tests, and while we can make do when starting with a fresh test list(an unrun test list has the story ids), we need to be able to feed data from test runs back into the split algorithm so that we can take advantage of test performance data to make the partitions perform equally.

The downside is that when a test fails, the test *id* won't specify the exact text file for stories (01-foo-bar.txt). OTOH we're removing those incrementally, and the filename in the detailed error page will be correct. So its not a big deal IMO.

To post a comment you must log in.
Revision history for this message
Jonathan Lange (jml) wrote :
Download full text (4.1 KiB)

On Tue, Oct 12, 2010 at 6:43 AM, Robert Collins
<email address hidden> wrote:
> Robert Collins has proposed merging lp:~lifeless/launchpad/doctest into lp:launchpad/devel.
>
> Requested reviews:
>  Launchpad code reviewers (launchpad-reviewers)
>
>
> Prep work for parallel testing. One of the main components of parallel testing is programmatic rearrangement of the tests to run, and story tests in particular were a problem.
>
> Previously the test *id* for the whole story was pagetest: relname. This is ok, but I've changed it to story- to make it more clear that this is for *stories*, not just pagetests.
>

You might be interested to know that some people insist on calling all
pagetests "stories". Nevertheless I take you to mean the silly
pagetests that must run in sequence (e.g. 01-foo.txt).

> And now the problem bit: the delegated tests were reporting their own ids; those ids don't exist (naturally, the story appearing as a single test so it can't be split hides the components) - and even if they did exist in the full test list, they can't be run separately.
>
> So this branch changes the ids of the individual doctests so that they all claim to be the story as far as subunit and testr is concerned. This makes the full test list able to be partitioned and parallel tested without skipping tests, and while we can make do when starting with a fresh test list(an unrun test list has the story ids), we need to be able to feed data from test runs back into the split algorithm so that we can take advantage of test performance data to make the partitions perform equally.
>
> The downside is that when a test fails, the test *id* won't specify the exact text file for stories (01-foo-bar.txt). OTOH we're removing those incrementally, and the filename in the detailed error page will be correct. So its not a big deal IMO.

This seems like a bearable solution.

> === modified file 'lib/canonical/launchpad/testing/pages.py'
> --- lib/canonical/launchpad/testing/pages.py    2010-10-04 20:46:55 +0000
> +++ lib/canonical/launchpad/testing/pages.py    2010-10-12 05:43:43 +0000
> @@ -848,7 +848,7 @@
>         return self._suite.countTestCases()
>
>     def shortDescription(self):
> -        return "pagetest: %s" % self._description
> +        return self._description
>
>     def id(self):
>         return self.shortDescription()
> @@ -910,20 +910,27 @@
>     numberedfilenames = sorted(numberedfilenames)
>     unnumberedfilenames = sorted(unnumberedfilenames)
>
> +    suite = unittest.TestSuite()
> +    checker = SpecialOutputChecker()
>     # Add unnumbered tests to the suite individually.
> -    checker = SpecialOutputChecker()
> -    suite = LayeredDocFileSuite(
> -        package=package, checker=checker, stdout_logging=False,
> -        layer=PageTestLayer, setUp=setUp,
> -        *[os.path.join(storydir, filename)
> -          for filename in unnumberedfilenames])
> +    if unnumberedfilenames:
> +        suite.addTest(LayeredDocFileSuite(
> +            package=package, checker=checker, stdout_logging=False,
> +            layer=PageTestLayer, setUp=setUp,
> +            *[os.path.join(storydir, filename)
> +              for filename in unnumberedfil...

Read more...

Revision history for this message
Jonathan Lange (jml) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/canonical/launchpad/mail/ftests/test_stub.py'
--- lib/canonical/launchpad/mail/ftests/test_stub.py 2010-10-04 19:50:45 +0000
+++ lib/canonical/launchpad/mail/ftests/test_stub.py 2010-10-15 20:47:45 +0000
@@ -5,7 +5,6 @@
55
6from doctest import DocTestSuite6from doctest import DocTestSuite
7import re7import re
8import unittest
98
10from zope.testing.renormalizing import RENormalizing9from zope.testing.renormalizing import RENormalizing
1110
@@ -114,6 +113,3 @@
114 (re.compile(r'Revision="\d+"'), 'Revision="1999"')]))113 (re.compile(r'Revision="\d+"'), 'Revision="1999"')]))
115 suite.layer = LaunchpadFunctionalLayer114 suite.layer = LaunchpadFunctionalLayer
116 return suite115 return suite
117
118if __name__ == '__main__':
119 unittest.main(test_suite())
120116
=== modified file 'lib/canonical/launchpad/mail/tests/test_commands.py'
--- lib/canonical/launchpad/mail/tests/test_commands.py 2010-08-20 20:31:18 +0000
+++ lib/canonical/launchpad/mail/tests/test_commands.py 2010-10-15 20:47:45 +0000
@@ -2,14 +2,7 @@
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4from doctest import DocTestSuite4from doctest import DocTestSuite
5import unittest
65
76
8def test_suite():7def test_suite():
9 suite = unittest.TestSuite()8 return DocTestSuite('canonical.launchpad.mail.commands')
10 suite.addTest(DocTestSuite('canonical.launchpad.mail.commands'))
11 return suite
12
13
14if __name__ == '__main__':
15 unittest.main(defaultTest='test_suite')
169
=== modified file 'lib/canonical/launchpad/mail/tests/test_handlers.py'
--- lib/canonical/launchpad/mail/tests/test_handlers.py 2010-10-04 19:50:45 +0000
+++ lib/canonical/launchpad/mail/tests/test_handlers.py 2010-10-15 20:47:45 +0000
@@ -114,7 +114,3 @@
114 suite.addTests(DocTestSuite('canonical.launchpad.mail.handlers'))114 suite.addTests(DocTestSuite('canonical.launchpad.mail.handlers'))
115 suite.addTests(unittest.TestLoader().loadTestsFromName(__name__))115 suite.addTests(unittest.TestLoader().loadTestsFromName(__name__))
116 return suite116 return suite
117
118
119if __name__ == '__main__':
120 unittest.main(defaultTest='test_suite')
121117
=== modified file 'lib/canonical/launchpad/mail/tests/test_helpers.py'
--- lib/canonical/launchpad/mail/tests/test_helpers.py 2010-10-04 19:50:45 +0000
+++ lib/canonical/launchpad/mail/tests/test_helpers.py 2010-10-15 20:47:45 +0000
@@ -223,7 +223,6 @@
223223
224224
225def test_suite():225def test_suite():
226 suite = unittest.TestSuite()226 suite = DocTestSuite('canonical.launchpad.mail.handlers')
227 suite.addTests(DocTestSuite('canonical.launchpad.mail.handlers'))
228 suite.addTests(unittest.TestLoader().loadTestsFromName(__name__))227 suite.addTests(unittest.TestLoader().loadTestsFromName(__name__))
229 return suite228 return suite
230229
=== modified file 'lib/canonical/launchpad/scripts/tests/test_rundoctests.py'
--- lib/canonical/launchpad/scripts/tests/test_rundoctests.py 2010-08-20 20:31:18 +0000
+++ lib/canonical/launchpad/scripts/tests/test_rundoctests.py 2010-10-15 20:47:45 +0000
@@ -19,7 +19,3 @@
19 'canonical.launchpad.scripts.logger', tearDown=tearDown19 'canonical.launchpad.scripts.logger', tearDown=tearDown
20 ))20 ))
21 return suite21 return suite
22
23if __name__ == '__main__':
24 unittest.main(defaultTest='test_suite')
25
2622
=== modified file 'lib/canonical/launchpad/testing/pages.py'
--- lib/canonical/launchpad/testing/pages.py 2010-10-04 20:46:55 +0000
+++ lib/canonical/launchpad/testing/pages.py 2010-10-15 20:47:45 +0000
@@ -848,7 +848,7 @@
848 return self._suite.countTestCases()848 return self._suite.countTestCases()
849849
850 def shortDescription(self):850 def shortDescription(self):
851 return "pagetest: %s" % self._description851 return self._description
852852
853 def id(self):853 def id(self):
854 return self.shortDescription()854 return self.shortDescription()
@@ -910,20 +910,27 @@
910 numberedfilenames = sorted(numberedfilenames)910 numberedfilenames = sorted(numberedfilenames)
911 unnumberedfilenames = sorted(unnumberedfilenames)911 unnumberedfilenames = sorted(unnumberedfilenames)
912912
913 suite = unittest.TestSuite()
914 checker = SpecialOutputChecker()
913 # Add unnumbered tests to the suite individually.915 # Add unnumbered tests to the suite individually.
914 checker = SpecialOutputChecker()916 if unnumberedfilenames:
915 suite = LayeredDocFileSuite(917 suite.addTest(LayeredDocFileSuite(
916 package=package, checker=checker, stdout_logging=False,918 package=package, checker=checker, stdout_logging=False,
917 layer=PageTestLayer, setUp=setUp,919 layer=PageTestLayer, setUp=setUp,
918 *[os.path.join(storydir, filename)920 *[os.path.join(storydir, filename)
919 for filename in unnumberedfilenames])921 for filename in unnumberedfilenames]))
920922
921 # Add numbered tests to the suite as a single story.923 # Add numbered tests to the suite as a single story.
922 storysuite = LayeredDocFileSuite(924 if numberedfilenames:
923 package=package, checker=checker, stdout_logging=False,925 storysuite = LayeredDocFileSuite(
924 setUp=setUp,926 package=package, checker=checker, stdout_logging=False,
925 *[os.path.join(storydir, filename)927 setUp=setUp,
926 for filename in numberedfilenames])928 *[os.path.join(storydir, filename)
927 suite.addTest(PageStoryTestCase(stripped_storydir, storysuite))929 for filename in numberedfilenames])
930 story_test_id = "story-%s" % stripped_storydir
931 get_id = lambda: story_test_id
932 for test in storysuite:
933 test.id = get_id
934 suite.addTest(PageStoryTestCase(story_test_id, storysuite))
928935
929 return suite936 return suite
930937
=== modified file 'lib/canonical/launchpad/testing/tests/test_pages.py'
--- lib/canonical/launchpad/testing/tests/test_pages.py 2010-10-04 19:50:45 +0000
+++ lib/canonical/launchpad/testing/tests/test_pages.py 2010-10-15 20:47:45 +0000
@@ -9,6 +9,8 @@
9import tempfile9import tempfile
10import unittest10import unittest
1111
12from bzrlib.tests import iter_suite_tests
13
12from canonical.launchpad.testing.pages import PageTestSuite14from canonical.launchpad.testing.pages import PageTestSuite
13from canonical.testing.layers import PageTestLayer15from canonical.testing.layers import PageTestLayer
1416
@@ -40,7 +42,7 @@
40 # module's path.42 # module's path.
41 suite = PageTestSuite(os.path.basename(self.tempdir))43 suite = PageTestSuite(os.path.basename(self.tempdir))
42 self.failUnless(isinstance(suite, unittest.TestSuite))44 self.failUnless(isinstance(suite, unittest.TestSuite))
43 [bar_test, story] = list(suite)45 bar_test, story = iter_suite_tests(suite)
4446
45 # The unnumbered file appears as an independent test.47 # The unnumbered file appears as an independent test.
46 self.assertEqual(os.path.basename(bar_test.id()), 'xx-bar_txt')48 self.assertEqual(os.path.basename(bar_test.id()), 'xx-bar_txt')
4749
=== modified file 'lib/canonical/launchpad/tests/test_datetimeutils.py'
--- lib/canonical/launchpad/tests/test_datetimeutils.py 2010-08-20 20:31:18 +0000
+++ lib/canonical/launchpad/tests/test_datetimeutils.py 2010-10-15 20:47:45 +0000
@@ -2,16 +2,9 @@
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4from doctest import DocTestSuite4from doctest import DocTestSuite
5import unittest
65
7from canonical.launchpad import datetimeutils6from canonical.launchpad import datetimeutils
87
98
10def test_suite():9def test_suite():
11 suite = unittest.TestSuite()10 return DocTestSuite(datetimeutils)
12 suite.addTest(DocTestSuite(datetimeutils))
13 return suite
14
15if __name__ == '__main__':
16 unittest.TextTestRunner().run(test_suite())
17
1811
=== modified file 'lib/canonical/launchpad/tests/test_helpers.py'
--- lib/canonical/launchpad/tests/test_helpers.py 2010-10-04 19:50:45 +0000
+++ lib/canonical/launchpad/tests/test_helpers.py 2010-10-15 20:47:45 +0000
@@ -319,7 +319,3 @@
319 suite.addTest(319 suite.addTest(
320 unittest.TestLoader().loadTestsFromTestCase(TestEmailPeople))320 unittest.TestLoader().loadTestsFromTestCase(TestEmailPeople))
321 return suite321 return suite
322
323if __name__ == '__main__':
324 unittest.TextTestRunner().run(test_suite())
325
326322
=== modified file 'lib/canonical/launchpad/webapp/tests/test_publisher.py'
--- lib/canonical/launchpad/webapp/tests/test_publisher.py 2010-08-20 20:31:18 +0000
+++ lib/canonical/launchpad/webapp/tests/test_publisher.py 2010-10-15 20:47:45 +0000
@@ -5,7 +5,6 @@
5 DocTestSuite,5 DocTestSuite,
6 ELLIPSIS,6 ELLIPSIS,
7 )7 )
8import unittest
98
10from canonical.launchpad.webapp import publisher9from canonical.launchpad.webapp import publisher
1110
@@ -13,7 +12,3 @@
13def test_suite():12def test_suite():
14 suite = DocTestSuite(publisher, optionflags=ELLIPSIS)13 suite = DocTestSuite(publisher, optionflags=ELLIPSIS)
15 return suite14 return suite
16
17
18if __name__ == '__main__':
19 unittest.main(defaultTest='test_suite')
2015
=== modified file 'lib/canonical/testing/tests/test_doctestcodec.py'
--- lib/canonical/testing/tests/test_doctestcodec.py 2010-07-14 14:11:15 +0000
+++ lib/canonical/testing/tests/test_doctestcodec.py 2010-10-15 20:47:45 +0000
@@ -7,12 +7,8 @@
7__all__ = []7__all__ = []
88
9from doctest import DocTestSuite9from doctest import DocTestSuite
10import unittest
1110
12import canonical.testing.doctestcodec11import canonical.testing.doctestcodec
1312
14def test_suite():13def test_suite():
15 suite = unittest.TestSuite()14 return DocTestSuite(canonical.testing.doctestcodec)
16 suite.addTest(DocTestSuite(canonical.testing.doctestcodec))
17 return suite
18
1915
=== modified file 'lib/canonical/tests/test_base.py'
--- lib/canonical/tests/test_base.py 2010-07-14 14:11:15 +0000
+++ lib/canonical/tests/test_base.py 2010-10-15 20:47:45 +0000
@@ -2,14 +2,8 @@
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4from doctest import DocTestSuite4from doctest import DocTestSuite
5import unittest
6import canonical.base5import canonical.base
76
8def test_suite():7def test_suite():
9 suite = DocTestSuite(canonical.base)8 suite = DocTestSuite(canonical.base)
10 return suite9 return suite
11
12if __name__ == "__main__":
13 DEFAULT = test_suite()
14 unittest.main(defaultTest='DEFAULT')
15
1610
=== modified file 'lib/canonical/tests/test_encoding.py'
--- lib/canonical/tests/test_encoding.py 2010-07-14 14:11:15 +0000
+++ lib/canonical/tests/test_encoding.py 2010-10-15 20:47:45 +0000
@@ -2,14 +2,9 @@
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4from doctest import DocTestSuite, ELLIPSIS4from doctest import DocTestSuite, ELLIPSIS
5import unittest
65
7import canonical.encoding6import canonical.encoding
87
9def test_suite():8def test_suite():
10 suite = DocTestSuite(canonical.encoding, optionflags=ELLIPSIS)9 suite = DocTestSuite(canonical.encoding, optionflags=ELLIPSIS)
11 return suite10 return suite
12
13
14if __name__ == '__main__':
15 unittest.main(defaultTest='test_suite')
1611
=== modified file 'lib/canonical/widgets/ftests/test_widget_doctests.py'
--- lib/canonical/widgets/ftests/test_widget_doctests.py 2010-10-04 19:50:45 +0000
+++ lib/canonical/widgets/ftests/test_widget_doctests.py 2010-10-15 20:47:45 +0000
@@ -13,7 +13,3 @@
13 suite.addTest(doctest.DocTestSuite('canonical.widgets.textwidgets'))13 suite.addTest(doctest.DocTestSuite('canonical.widgets.textwidgets'))
14 suite.addTest(doctest.DocTestSuite('canonical.widgets.date'))14 suite.addTest(doctest.DocTestSuite('canonical.widgets.date'))
15 return suite15 return suite
16
17if __name__ == '__main__':
18 default_test = test_suite()
19 unittest.main('default_test')
2016
=== modified file 'lib/lp/services/mail/tests/test_incoming.py'
--- lib/lp/services/mail/tests/test_incoming.py 2010-10-12 14:20:44 +0000
+++ lib/lp/services/mail/tests/test_incoming.py 2010-10-15 20:47:45 +0000
@@ -7,7 +7,6 @@
7import unittest7import unittest
88
9import transaction9import transaction
10
11from zope.security.management import setSecurityPolicy10from zope.security.management import setSecurityPolicy
1211
13from canonical.config import config12from canonical.config import config
@@ -93,8 +92,7 @@
9392
9493
95def test_suite():94def test_suite():
96 suite = unittest.TestSuite()95 suite = unittest.TestLoader().loadTestsFromName(__name__)
97 suite.addTest(unittest.TestLoader().loadTestsFromName(__name__))
98 suite.addTest(DocTestSuite('lp.services.mail.incoming'))96 suite.addTest(DocTestSuite('lp.services.mail.incoming'))
99 suite.addTest(97 suite.addTest(
100 LayeredDocFileSuite(98 LayeredDocFileSuite(
@@ -104,7 +102,3 @@
104 layer=LaunchpadZopelessLayer,102 layer=LaunchpadZopelessLayer,
105 stdout_logging_level=logging.WARNING))103 stdout_logging_level=logging.WARNING))
106 return suite104 return suite
107
108
109if __name__ == '__main__':
110 unittest.main(defaultTest='test_suite')