Launchpad itself

Merge lp:~lifeless/launchpad/doctest into lp:launchpad

  1. doctest
  2. Merge into devel
Proposed by Robert Collins
Status: Merged
Approved by: Jonathan Lange
Approved revision: no longer in the source branch.
Merged at revision: 11723
Proposed branch: lp:~lifeless/launchpad/doctest
Merge into: lp:launchpad
Diff against target: 302 lines (+28/-80)
15 files modified
lib/canonical/launchpad/mail/ftests/test_stub.py (+0/-4)
lib/canonical/launchpad/mail/tests/test_commands.py (+1/-8)
lib/canonical/launchpad/mail/tests/test_handlers.py (+0/-4)
lib/canonical/launchpad/mail/tests/test_helpers.py (+1/-2)
lib/canonical/launchpad/scripts/tests/test_rundoctests.py (+0/-4)
lib/canonical/launchpad/testing/pages.py (+20/-13)
lib/canonical/launchpad/testing/tests/test_pages.py (+3/-1)
lib/canonical/launchpad/tests/test_datetimeutils.py (+1/-8)
lib/canonical/launchpad/tests/test_helpers.py (+0/-4)
lib/canonical/launchpad/webapp/tests/test_publisher.py (+0/-5)
lib/canonical/testing/tests/test_doctestcodec.py (+1/-5)
lib/canonical/tests/test_base.py (+0/-6)
lib/canonical/tests/test_encoding.py (+0/-5)
lib/canonical/widgets/ftests/test_widget_doctests.py (+0/-4)
lib/lp/services/mail/tests/test_incoming.py (+1/-7)
To merge this branch: bzr merge lp:~lifeless/launchpad/doctest
Reviewer Review Type Date Requested Status
Jonathan Lange (community) Approve
Review via email: mp+38182@code.launchpad.net

Commit message

Change the reported test id for all elements of a story to be the id of the story.

Description of the change

Prep work for parallel testing. One of the main components of parallel testing is programmatic rearrangement of the tests to run, and story tests in particular were a problem.

Previously the test *id* for the whole story was pagetest: relname. This is ok, but I've changed it to story- to make it more clear that this is for *stories*, not just pagetests.

And now the problem bit: the delegated tests were reporting their own ids; those ids don't exist (naturally, the story appearing as a single test so it can't be split hides the components) - and even if they did exist in the full test list, they can't be run separately.

So this branch changes the ids of the individual doctests so that they all claim to be the story as far as subunit and testr is concerned. This makes the full test list able to be partitioned and parallel tested without skipping tests, and while we can make do when starting with a fresh test list(an unrun test list has the story ids), we need to be able to feed data from test runs back into the split algorithm so that we can take advantage of test performance data to make the partitions perform equally.

The downside is that when a test fails, the test *id* won't specify the exact text file for stories (01-foo-bar.txt). OTOH we're removing those incrementally, and the filename in the detailed error page will be correct. So its not a big deal IMO.

To post a comment you must log in.
Revision history for this message
Jonathan Lange (jml) wrote :
Download full text (4.1 KiB)

On Tue, Oct 12, 2010 at 6:43 AM, Robert Collins
<email address hidden> wrote:
> Robert Collins has proposed merging lp:~lifeless/launchpad/doctest into lp:launchpad/devel.
>
> Requested reviews:
>  Launchpad code reviewers (launchpad-reviewers)
>
>
> Prep work for parallel testing. One of the main components of parallel testing is programmatic rearrangement of the tests to run, and story tests in particular were a problem.
>
> Previously the test *id* for the whole story was pagetest: relname. This is ok, but I've changed it to story- to make it more clear that this is for *stories*, not just pagetests.
>

You might be interested to know that some people insist on calling all
pagetests "stories". Nevertheless I take you to mean the silly
pagetests that must run in sequence (e.g. 01-foo.txt).

> And now the problem bit: the delegated tests were reporting their own ids; those ids don't exist (naturally, the story appearing as a single test so it can't be split hides the components) - and even if they did exist in the full test list, they can't be run separately.
>
> So this branch changes the ids of the individual doctests so that they all claim to be the story as far as subunit and testr is concerned. This makes the full test list able to be partitioned and parallel tested without skipping tests, and while we can make do when starting with a fresh test list(an unrun test list has the story ids), we need to be able to feed data from test runs back into the split algorithm so that we can take advantage of test performance data to make the partitions perform equally.
>
> The downside is that when a test fails, the test *id* won't specify the exact text file for stories (01-foo-bar.txt). OTOH we're removing those incrementally, and the filename in the detailed error page will be correct. So its not a big deal IMO.

This seems like a bearable solution.

> === modified file 'lib/canonical/launchpad/testing/pages.py'
> --- lib/canonical/launchpad/testing/pages.py    2010-10-04 20:46:55 +0000
> +++ lib/canonical/launchpad/testing/pages.py    2010-10-12 05:43:43 +0000
> @@ -848,7 +848,7 @@
>         return self._suite.countTestCases()
>
>     def shortDescription(self):
> -        return "pagetest: %s" % self._description
> +        return self._description
>
>     def id(self):
>         return self.shortDescription()
> @@ -910,20 +910,27 @@
>     numberedfilenames = sorted(numberedfilenames)
>     unnumberedfilenames = sorted(unnumberedfilenames)
>
> +    suite = unittest.TestSuite()
> +    checker = SpecialOutputChecker()
>     # Add unnumbered tests to the suite individually.
> -    checker = SpecialOutputChecker()
> -    suite = LayeredDocFileSuite(
> -        package=package, checker=checker, stdout_logging=False,
> -        layer=PageTestLayer, setUp=setUp,
> -        *[os.path.join(storydir, filename)
> -          for filename in unnumberedfilenames])
> +    if unnumberedfilenames:
> +        suite.addTest(LayeredDocFileSuite(
> +            package=package, checker=checker, stdout_logging=False,
> +            layer=PageTestLayer, setUp=setUp,
> +            *[os.path.join(storydir, filename)
> +              for filename in unnumberedfil...

Read more...

Revision history for this message
Jonathan Lange (jml) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/launchpad/mail/ftests/test_stub.py'
2--- lib/canonical/launchpad/mail/ftests/test_stub.py 2010-10-04 19:50:45 +0000
3+++ lib/canonical/launchpad/mail/ftests/test_stub.py 2010-10-15 20:47:45 +0000
4@@ -5,7 +5,6 @@
5
6 from doctest import DocTestSuite
7 import re
8-import unittest
9
10 from zope.testing.renormalizing import RENormalizing
11
12@@ -114,6 +113,3 @@
13 (re.compile(r'Revision="\d+"'), 'Revision="1999"')]))
14 suite.layer = LaunchpadFunctionalLayer
15 return suite
16-
17-if __name__ == '__main__':
18- unittest.main(test_suite())
19
20=== modified file 'lib/canonical/launchpad/mail/tests/test_commands.py'
21--- lib/canonical/launchpad/mail/tests/test_commands.py 2010-08-20 20:31:18 +0000
22+++ lib/canonical/launchpad/mail/tests/test_commands.py 2010-10-15 20:47:45 +0000
23@@ -2,14 +2,7 @@
24 # GNU Affero General Public License version 3 (see the file LICENSE).
25
26 from doctest import DocTestSuite
27-import unittest
28
29
30 def test_suite():
31- suite = unittest.TestSuite()
32- suite.addTest(DocTestSuite('canonical.launchpad.mail.commands'))
33- return suite
34-
35-
36-if __name__ == '__main__':
37- unittest.main(defaultTest='test_suite')
38+ return DocTestSuite('canonical.launchpad.mail.commands')
39
40=== modified file 'lib/canonical/launchpad/mail/tests/test_handlers.py'
41--- lib/canonical/launchpad/mail/tests/test_handlers.py 2010-10-04 19:50:45 +0000
42+++ lib/canonical/launchpad/mail/tests/test_handlers.py 2010-10-15 20:47:45 +0000
43@@ -114,7 +114,3 @@
44 suite.addTests(DocTestSuite('canonical.launchpad.mail.handlers'))
45 suite.addTests(unittest.TestLoader().loadTestsFromName(__name__))
46 return suite
47-
48-
49-if __name__ == '__main__':
50- unittest.main(defaultTest='test_suite')
51
52=== modified file 'lib/canonical/launchpad/mail/tests/test_helpers.py'
53--- lib/canonical/launchpad/mail/tests/test_helpers.py 2010-10-04 19:50:45 +0000
54+++ lib/canonical/launchpad/mail/tests/test_helpers.py 2010-10-15 20:47:45 +0000
55@@ -223,7 +223,6 @@
56
57
58 def test_suite():
59- suite = unittest.TestSuite()
60- suite.addTests(DocTestSuite('canonical.launchpad.mail.handlers'))
61+ suite = DocTestSuite('canonical.launchpad.mail.handlers')
62 suite.addTests(unittest.TestLoader().loadTestsFromName(__name__))
63 return suite
64
65=== modified file 'lib/canonical/launchpad/scripts/tests/test_rundoctests.py'
66--- lib/canonical/launchpad/scripts/tests/test_rundoctests.py 2010-08-20 20:31:18 +0000
67+++ lib/canonical/launchpad/scripts/tests/test_rundoctests.py 2010-10-15 20:47:45 +0000
68@@ -19,7 +19,3 @@
69 'canonical.launchpad.scripts.logger', tearDown=tearDown
70 ))
71 return suite
72-
73-if __name__ == '__main__':
74- unittest.main(defaultTest='test_suite')
75-
76
77=== modified file 'lib/canonical/launchpad/testing/pages.py'
78--- lib/canonical/launchpad/testing/pages.py 2010-10-04 20:46:55 +0000
79+++ lib/canonical/launchpad/testing/pages.py 2010-10-15 20:47:45 +0000
80@@ -848,7 +848,7 @@
81 return self._suite.countTestCases()
82
83 def shortDescription(self):
84- return "pagetest: %s" % self._description
85+ return self._description
86
87 def id(self):
88 return self.shortDescription()
89@@ -910,20 +910,27 @@
90 numberedfilenames = sorted(numberedfilenames)
91 unnumberedfilenames = sorted(unnumberedfilenames)
92
93+ suite = unittest.TestSuite()
94+ checker = SpecialOutputChecker()
95 # Add unnumbered tests to the suite individually.
96- checker = SpecialOutputChecker()
97- suite = LayeredDocFileSuite(
98- package=package, checker=checker, stdout_logging=False,
99- layer=PageTestLayer, setUp=setUp,
100- *[os.path.join(storydir, filename)
101- for filename in unnumberedfilenames])
102+ if unnumberedfilenames:
103+ suite.addTest(LayeredDocFileSuite(
104+ package=package, checker=checker, stdout_logging=False,
105+ layer=PageTestLayer, setUp=setUp,
106+ *[os.path.join(storydir, filename)
107+ for filename in unnumberedfilenames]))
108
109 # Add numbered tests to the suite as a single story.
110- storysuite = LayeredDocFileSuite(
111- package=package, checker=checker, stdout_logging=False,
112- setUp=setUp,
113- *[os.path.join(storydir, filename)
114- for filename in numberedfilenames])
115- suite.addTest(PageStoryTestCase(stripped_storydir, storysuite))
116+ if numberedfilenames:
117+ storysuite = LayeredDocFileSuite(
118+ package=package, checker=checker, stdout_logging=False,
119+ setUp=setUp,
120+ *[os.path.join(storydir, filename)
121+ for filename in numberedfilenames])
122+ story_test_id = "story-%s" % stripped_storydir
123+ get_id = lambda: story_test_id
124+ for test in storysuite:
125+ test.id = get_id
126+ suite.addTest(PageStoryTestCase(story_test_id, storysuite))
127
128 return suite
129
130=== modified file 'lib/canonical/launchpad/testing/tests/test_pages.py'
131--- lib/canonical/launchpad/testing/tests/test_pages.py 2010-10-04 19:50:45 +0000
132+++ lib/canonical/launchpad/testing/tests/test_pages.py 2010-10-15 20:47:45 +0000
133@@ -9,6 +9,8 @@
134 import tempfile
135 import unittest
136
137+from bzrlib.tests import iter_suite_tests
138+
139 from canonical.launchpad.testing.pages import PageTestSuite
140 from canonical.testing.layers import PageTestLayer
141
142@@ -40,7 +42,7 @@
143 # module's path.
144 suite = PageTestSuite(os.path.basename(self.tempdir))
145 self.failUnless(isinstance(suite, unittest.TestSuite))
146- [bar_test, story] = list(suite)
147+ bar_test, story = iter_suite_tests(suite)
148
149 # The unnumbered file appears as an independent test.
150 self.assertEqual(os.path.basename(bar_test.id()), 'xx-bar_txt')
151
152=== modified file 'lib/canonical/launchpad/tests/test_datetimeutils.py'
153--- lib/canonical/launchpad/tests/test_datetimeutils.py 2010-08-20 20:31:18 +0000
154+++ lib/canonical/launchpad/tests/test_datetimeutils.py 2010-10-15 20:47:45 +0000
155@@ -2,16 +2,9 @@
156 # GNU Affero General Public License version 3 (see the file LICENSE).
157
158 from doctest import DocTestSuite
159-import unittest
160
161 from canonical.launchpad import datetimeutils
162
163
164 def test_suite():
165- suite = unittest.TestSuite()
166- suite.addTest(DocTestSuite(datetimeutils))
167- return suite
168-
169-if __name__ == '__main__':
170- unittest.TextTestRunner().run(test_suite())
171-
172+ return DocTestSuite(datetimeutils)
173
174=== modified file 'lib/canonical/launchpad/tests/test_helpers.py'
175--- lib/canonical/launchpad/tests/test_helpers.py 2010-10-04 19:50:45 +0000
176+++ lib/canonical/launchpad/tests/test_helpers.py 2010-10-15 20:47:45 +0000
177@@ -319,7 +319,3 @@
178 suite.addTest(
179 unittest.TestLoader().loadTestsFromTestCase(TestEmailPeople))
180 return suite
181-
182-if __name__ == '__main__':
183- unittest.TextTestRunner().run(test_suite())
184-
185
186=== modified file 'lib/canonical/launchpad/webapp/tests/test_publisher.py'
187--- lib/canonical/launchpad/webapp/tests/test_publisher.py 2010-08-20 20:31:18 +0000
188+++ lib/canonical/launchpad/webapp/tests/test_publisher.py 2010-10-15 20:47:45 +0000
189@@ -5,7 +5,6 @@
190 DocTestSuite,
191 ELLIPSIS,
192 )
193-import unittest
194
195 from canonical.launchpad.webapp import publisher
196
197@@ -13,7 +12,3 @@
198 def test_suite():
199 suite = DocTestSuite(publisher, optionflags=ELLIPSIS)
200 return suite
201-
202-
203-if __name__ == '__main__':
204- unittest.main(defaultTest='test_suite')
205
206=== modified file 'lib/canonical/testing/tests/test_doctestcodec.py'
207--- lib/canonical/testing/tests/test_doctestcodec.py 2010-07-14 14:11:15 +0000
208+++ lib/canonical/testing/tests/test_doctestcodec.py 2010-10-15 20:47:45 +0000
209@@ -7,12 +7,8 @@
210 __all__ = []
211
212 from doctest import DocTestSuite
213-import unittest
214
215 import canonical.testing.doctestcodec
216
217 def test_suite():
218- suite = unittest.TestSuite()
219- suite.addTest(DocTestSuite(canonical.testing.doctestcodec))
220- return suite
221-
222+ return DocTestSuite(canonical.testing.doctestcodec)
223
224=== modified file 'lib/canonical/tests/test_base.py'
225--- lib/canonical/tests/test_base.py 2010-07-14 14:11:15 +0000
226+++ lib/canonical/tests/test_base.py 2010-10-15 20:47:45 +0000
227@@ -2,14 +2,8 @@
228 # GNU Affero General Public License version 3 (see the file LICENSE).
229
230 from doctest import DocTestSuite
231-import unittest
232 import canonical.base
233
234 def test_suite():
235 suite = DocTestSuite(canonical.base)
236 return suite
237-
238-if __name__ == "__main__":
239- DEFAULT = test_suite()
240- unittest.main(defaultTest='DEFAULT')
241-
242
243=== modified file 'lib/canonical/tests/test_encoding.py'
244--- lib/canonical/tests/test_encoding.py 2010-07-14 14:11:15 +0000
245+++ lib/canonical/tests/test_encoding.py 2010-10-15 20:47:45 +0000
246@@ -2,14 +2,9 @@
247 # GNU Affero General Public License version 3 (see the file LICENSE).
248
249 from doctest import DocTestSuite, ELLIPSIS
250-import unittest
251
252 import canonical.encoding
253
254 def test_suite():
255 suite = DocTestSuite(canonical.encoding, optionflags=ELLIPSIS)
256 return suite
257-
258-
259-if __name__ == '__main__':
260- unittest.main(defaultTest='test_suite')
261
262=== modified file 'lib/canonical/widgets/ftests/test_widget_doctests.py'
263--- lib/canonical/widgets/ftests/test_widget_doctests.py 2010-10-04 19:50:45 +0000
264+++ lib/canonical/widgets/ftests/test_widget_doctests.py 2010-10-15 20:47:45 +0000
265@@ -13,7 +13,3 @@
266 suite.addTest(doctest.DocTestSuite('canonical.widgets.textwidgets'))
267 suite.addTest(doctest.DocTestSuite('canonical.widgets.date'))
268 return suite
269-
270-if __name__ == '__main__':
271- default_test = test_suite()
272- unittest.main('default_test')
273
274=== modified file 'lib/lp/services/mail/tests/test_incoming.py'
275--- lib/lp/services/mail/tests/test_incoming.py 2010-10-12 14:20:44 +0000
276+++ lib/lp/services/mail/tests/test_incoming.py 2010-10-15 20:47:45 +0000
277@@ -7,7 +7,6 @@
278 import unittest
279
280 import transaction
281-
282 from zope.security.management import setSecurityPolicy
283
284 from canonical.config import config
285@@ -93,8 +92,7 @@
286
287
288 def test_suite():
289- suite = unittest.TestSuite()
290- suite.addTest(unittest.TestLoader().loadTestsFromName(__name__))
291+ suite = unittest.TestLoader().loadTestsFromName(__name__)
292 suite.addTest(DocTestSuite('lp.services.mail.incoming'))
293 suite.addTest(
294 LayeredDocFileSuite(
295@@ -104,7 +102,3 @@
296 layer=LaunchpadZopelessLayer,
297 stdout_logging_level=logging.WARNING))
298 return suite
299-
300-
301-if __name__ == '__main__':
302- unittest.main(defaultTest='test_suite')