Bazaar

Merge lp:~mbp/bzr/597791-http-tests into lp:bzr

  1. 597791-http-tests
  2. Merge into bzr.dev
Proposed by Martin Pool
Status: Merged
Approved by: Vincent Ladeuil
Approved revision: no longer in the source branch.
Merged at revision: 5483
Proposed branch: lp:~mbp/bzr/597791-http-tests
Merge into: lp:bzr
Diff against target: 689 lines (+326/-100)
6 files modified
NEWS (+6/-0)
bzrlib/tests/__init__.py (+14/-1)
bzrlib/tests/scenarios.py (+61/-0)
bzrlib/tests/test_http.py (+115/-89)
bzrlib/tests/test_scenarios.py (+110/-0)
doc/developers/testing.txt (+20/-10)
To merge this branch: bzr merge lp:~mbp/bzr/597791-http-tests
Reviewer Review Type Date Requested Status
Robert Collins (community) Approve
Vincent Ladeuil Approve
Review via email: mp+37941@code.launchpad.net

Commit message

cleanup test_http to use scenarios; add load_tests_from_scenarios

Description of the change

TestCase classes can now just have a `variations` attribute to describe how they should be multiplied. This lets us entirely delete the load_tests function complained of in bug 597791 and to my mind it makes the code _much_ nicer by eliminating spooky action at a distance. Now each class says what kind of variation it wants, without needing to know all the specific values, which may be dynamically determined.

I tested this doesn't change the parameterization by running 'selftest --list' on this and trunk, and ignoring ordering they're the same.

It could be nice to move or copy this into testscenarios.

I haven't yet rolled this out to other modules but I expect in many cases we could delete their custom load_tests methods in favour of just using load_tests_from_their_variations. (The name is a bit explicit because the behaviour's a bit magical.) I think this may also help with cases where we've used subclassing to parameterize because the official way was a bit hard.

To post a comment you must log in.
Revision history for this message
Martin Pool (mbp) wrote :

After some discussion with Robert and bug 658044, I think I can make this converge a bit more with what's already done in testscenarios:

 * name the attribute 'scenarios'
 * make it an iterable of scenarios directly, rather than a list of TestVariations to multiply
 * generate the cross product by calling a function rather than implicitly; to work well with scenario generators that come from registries this function should be quite lazy
 * change the variations to generators or functions rather than classes

Also stylistically
 * clearer tests
 * remove extra VWS
 * module docstring

Revision history for this message
Vincent Ladeuil (vila) wrote :

Lovely result :)

I had a look at that strange assertionError and... it looks like an uncaught regression *long* ago.
Or even worse, this looks like a test that has *never* failed :-(
I'll look at it again tomorrow.

One question: why is it required to do:

  load_tests = load_tests_from_scenarios

Can't that be triggered automatically if a 'scenarios' attribute is defined on the test class ?

It's true that load_tests is generally required when you want to parametrize but... is it still possible to define a load_tests function and call load_tests_from_scenarios for some classes only ?

I can't put my finger on it as I write this but I somehow feel there is a hole somewhere... imbw, you may just have fill the hole instead...

review: Approve
Revision history for this message
Martin Pool (mbp) wrote :

I added (obviously) the raise AssertionError() just to make it really obvious that the thing does indeed fail. My guess is that an error here is sent back to the client as an http 500 error, and it accepts that as a reasonable response.

Is load_tests needed? I think Robert added that as a semi-standard extension point where the test loader calls back in to our code. If we want the test suite to work with other runners, we can't hardcode a check for scenarios. Perhaps we can simplify it more. otoh explicit is better than implicit, and now it's only one line (plus an import...)

Revision history for this message
Robert Collins (lifeless) wrote :

load_tests is in python 2.7 (with a slightly different signature); it would be nice to adapt to that.

Doing load_tests = load_tests_from_scenarios (perhaps it would be better to say 'load_tests_apply_scenarios', but - *blue*) is pretty pithy, explicit and not terrible.

Doing a global is fine to - you could simply use generate_scenarios in bzrlib.test_suite()

381 + # XXX: this is strange; the 'random' name below seems undefined and
382 + # yet the tests pass -- mbp 2010-10-11
383 + raise AssertionError()

Looks like you can't land it like this?

This looks lovely. I would love to see a patch with the missing bits put into testscenarios.

review: Approve
Revision history for this message
Martin Pool (mbp) wrote :

> load_tests is in python 2.7 (with a slightly different signature); it would be
> nice to adapt to that.

Right, bug 607412.

> Doing load_tests = load_tests_from_scenarios (perhaps it would be better to
> say 'load_tests_apply_scenarios', but - *blue*) is pretty pithy, explicit and
> not terrible.
>
> Doing a global is fine to - you could simply use generate_scenarios in
> bzrlib.test_suite()
>
>
> 381 + # XXX: this is strange; the 'random' name below seems
> undefined and
> 382 + # yet the tests pass -- mbp 2010-10-11
> 383 + raise AssertionError()
>
> Looks like you can't land it like this?

I can; the bug (bug 658773) was latent before I came here, and I just noticed it by running pyflakes.

>
> This looks lovely. I would love to see a patch with the missing bits put into
> testscenarios.

Right, I'll do that under bug 658044.

Revision history for this message
Martin Pool (mbp) wrote :

sent to pqm by email

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'NEWS'
2--- NEWS 2010-10-11 15:18:38 +0000
3+++ NEWS 2010-10-11 22:43:41 +0000
4@@ -158,6 +158,12 @@
5 Testing
6 *******
7
8+* Add a new simpler way to generate multiple test variations, by setting
9+ the `scenarios` attribute of a test class to a list of scenarios
10+ descriptions, then using `load_tests_apply_scenarios`. (See the testing
11+ guide and `bzrlib.tests.scenarios`.) Simplify `test_http` using this.
12+ (Martin Pool, #597791)
13+
14 * Add ``tests/ssl_certs/ca.crt`` to the required test files list. Test
15 involving the pycurl https test server fail otherwise when running
16 selftest from an installed version. (Vincent Ladeuil, #651706)
17
18=== modified file 'bzrlib/tests/__init__.py'
19--- bzrlib/tests/__init__.py 2010-10-08 10:16:12 +0000
20+++ bzrlib/tests/__init__.py 2010-10-11 22:43:41 +0000
21@@ -3809,6 +3809,7 @@
22 'bzrlib.tests.test_rio',
23 'bzrlib.tests.test_rules',
24 'bzrlib.tests.test_sampler',
25+ 'bzrlib.tests.test_scenarios',
26 'bzrlib.tests.test_script',
27 'bzrlib.tests.test_selftest',
28 'bzrlib.tests.test_serializer',
29@@ -3991,7 +3992,19 @@
30 return suite
31
32
33-def multiply_scenarios(scenarios_left, scenarios_right):
34+def multiply_scenarios(*scenarios):
35+ """Multiply two or more iterables of scenarios.
36+
37+ It is safe to pass scenario generators or iterators.
38+
39+ :returns: A list of compound scenarios: the cross-product of all
40+ scenarios, with the names concatenated and the parameters
41+ merged together.
42+ """
43+ return reduce(_multiply_two_scenarios, map(list, scenarios))
44+
45+
46+def _multiply_two_scenarios(scenarios_left, scenarios_right):
47 """Multiply two sets of scenarios.
48
49 :returns: the cartesian product of the two sets of scenarios, that is
50
51=== added file 'bzrlib/tests/scenarios.py'
52--- bzrlib/tests/scenarios.py 1970-01-01 00:00:00 +0000
53+++ bzrlib/tests/scenarios.py 2010-10-11 22:43:41 +0000
54@@ -0,0 +1,61 @@
55+# Copyright (C) 2005-2010 Canonical Ltd
56+#
57+# This program is free software; you can redistribute it and/or modify
58+# it under the terms of the GNU General Public License as published by
59+# the Free Software Foundation; either version 2 of the License, or
60+# (at your option) any later version.
61+#
62+# This program is distributed in the hope that it will be useful,
63+# but WITHOUT ANY WARRANTY; without even the implied warranty of
64+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
65+# GNU General Public License for more details.
66+#
67+# You should have received a copy of the GNU General Public License
68+# along with this program; if not, write to the Free Software
69+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
70+
71+
72+"""Generate multiple variations in different scenarios.
73+
74+For a class whose tests should be repeated in varying scenarios, set a
75+`scenarios` member to a list of scenarios where it should be repeated.
76+
77+This is similar to the interface provided by
78+<http://launchpad.net/testscenarios/>.
79+"""
80+
81+
82+from bzrlib.tests import (
83+ iter_suite_tests,
84+ multiply_scenarios,
85+ multiply_tests,
86+ )
87+
88+
89+def load_tests_apply_scenarios(standard_tests, module, loader):
90+ """Multiply tests depending on their 'scenarios' attribute.
91+
92+ This can be assigned to 'load_tests' in any test module to make this
93+ automatically work across tests in the module.
94+ """
95+ result = loader.suiteClass()
96+ multiply_tests_by_their_scenarios(standard_tests, result)
97+ return result
98+
99+
100+def multiply_tests_by_their_scenarios(some_tests, into_suite):
101+ """Multiply the tests in the given suite by their declared scenarios.
102+
103+ Each test must have a 'scenarios' attribute which is a list of
104+ (name, params) pairs.
105+
106+ :param some_tests: TestSuite or Test.
107+ :param into_suite: A TestSuite into which the resulting tests will be
108+ inserted.
109+ """
110+ for test in iter_suite_tests(some_tests):
111+ scenarios = getattr(test, 'scenarios', None)
112+ if scenarios is None:
113+ into_suite.addTest(test)
114+ else:
115+ multiply_tests(test, test.scenarios, into_suite)
116
117=== modified file 'bzrlib/tests/test_http.py'
118--- bzrlib/tests/test_http.py 2010-08-24 13:03:18 +0000
119+++ bzrlib/tests/test_http.py 2010-10-11 22:43:41 +0000
120@@ -23,10 +23,7 @@
121 # TODO: Should be renamed to bzrlib.transport.http.tests?
122 # TODO: What about renaming to bzrlib.tests.transport.http ?
123
124-from cStringIO import StringIO
125 import httplib
126-import os
127-import select
128 import SimpleHTTPServer
129 import socket
130 import sys
131@@ -42,7 +39,6 @@
132 tests,
133 transport,
134 ui,
135- urlutils,
136 )
137 from bzrlib.tests import (
138 features,
139@@ -50,6 +46,10 @@
140 http_utils,
141 test_server,
142 )
143+from bzrlib.tests.scenarios import (
144+ load_tests_apply_scenarios,
145+ multiply_scenarios,
146+ )
147 from bzrlib.transport import (
148 http,
149 remote,
150@@ -64,17 +64,11 @@
151 from bzrlib.transport.http._pycurl import PyCurlTransport
152
153
154-def load_tests(standard_tests, module, loader):
155- """Multiply tests for http clients and protocol versions."""
156- result = loader.suiteClass()
157-
158- # one for each transport implementation
159- t_tests, remaining_tests = tests.split_suite_by_condition(
160- standard_tests, tests.condition_isinstance((
161- TestHttpTransportRegistration,
162- TestHttpTransportUrls,
163- Test_redirected_to,
164- )))
165+load_tests = load_tests_apply_scenarios
166+
167+
168+def vary_by_http_client_implementation():
169+ """Test the two libraries we can use, pycurl and urllib."""
170 transport_scenarios = [
171 ('urllib', dict(_transport=_urllib.HttpTransport_urllib,
172 _server=http_server.HttpServer_urllib,
173@@ -85,85 +79,48 @@
174 ('pycurl', dict(_transport=PyCurlTransport,
175 _server=http_server.HttpServer_PyCurl,
176 _url_protocol='http+pycurl',)))
177- tests.multiply_tests(t_tests, transport_scenarios, result)
178-
179- protocol_scenarios = [
180- ('HTTP/1.0', dict(_protocol_version='HTTP/1.0')),
181- ('HTTP/1.1', dict(_protocol_version='HTTP/1.1')),
182- ]
183-
184- # some tests are parametrized by the protocol version only
185- p_tests, remaining_tests = tests.split_suite_by_condition(
186- remaining_tests, tests.condition_isinstance((
187- TestAuthOnRedirected,
188- )))
189- tests.multiply_tests(p_tests, protocol_scenarios, result)
190-
191- # each implementation tested with each HTTP version
192- tp_tests, remaining_tests = tests.split_suite_by_condition(
193- remaining_tests, tests.condition_isinstance((
194- SmartHTTPTunnellingTest,
195- TestDoCatchRedirections,
196- TestHTTPConnections,
197- TestHTTPRedirections,
198- TestHTTPSilentRedirections,
199- TestLimitedRangeRequestServer,
200- TestPost,
201- TestProxyHttpServer,
202- TestRanges,
203- TestSpecificRequestHandler,
204- )))
205- tp_scenarios = tests.multiply_scenarios(transport_scenarios,
206- protocol_scenarios)
207- tests.multiply_tests(tp_tests, tp_scenarios, result)
208-
209- # proxy auth: each auth scheme on all http versions on all implementations.
210- tppa_tests, remaining_tests = tests.split_suite_by_condition(
211- remaining_tests, tests.condition_isinstance((
212- TestProxyAuth,
213- )))
214- proxy_auth_scheme_scenarios = [
215+ return transport_scenarios
216+
217+
218+def vary_by_http_protocol_version():
219+ """Test on http/1.0 and 1.1"""
220+ return [
221+ ('HTTP/1.0', dict(_protocol_version='HTTP/1.0')),
222+ ('HTTP/1.1', dict(_protocol_version='HTTP/1.1')),
223+ ]
224+
225+
226+def vary_by_http_proxy_auth_scheme():
227+ return [
228 ('basic', dict(_auth_server=http_utils.ProxyBasicAuthServer)),
229 ('digest', dict(_auth_server=http_utils.ProxyDigestAuthServer)),
230 ('basicdigest',
231- dict(_auth_server=http_utils.ProxyBasicAndDigestAuthServer)),
232+ dict(_auth_server=http_utils.ProxyBasicAndDigestAuthServer)),
233 ]
234- tppa_scenarios = tests.multiply_scenarios(tp_scenarios,
235- proxy_auth_scheme_scenarios)
236- tests.multiply_tests(tppa_tests, tppa_scenarios, result)
237-
238- # auth: each auth scheme on all http versions on all implementations.
239- tpa_tests, remaining_tests = tests.split_suite_by_condition(
240- remaining_tests, tests.condition_isinstance((
241- TestAuth,
242- )))
243- auth_scheme_scenarios = [
244+
245+
246+def vary_by_http_auth_scheme():
247+ return [
248 ('basic', dict(_auth_server=http_utils.HTTPBasicAuthServer)),
249 ('digest', dict(_auth_server=http_utils.HTTPDigestAuthServer)),
250 ('basicdigest',
251- dict(_auth_server=http_utils.HTTPBasicAndDigestAuthServer)),
252+ dict(_auth_server=http_utils.HTTPBasicAndDigestAuthServer)),
253 ]
254- tpa_scenarios = tests.multiply_scenarios(tp_scenarios,
255- auth_scheme_scenarios)
256- tests.multiply_tests(tpa_tests, tpa_scenarios, result)
257-
258- # activity: on all http[s] versions on all implementations
259- tpact_tests, remaining_tests = tests.split_suite_by_condition(
260- remaining_tests, tests.condition_isinstance((
261- TestActivity,
262- )))
263+
264+
265+def vary_by_http_activity():
266 activity_scenarios = [
267 ('urllib,http', dict(_activity_server=ActivityHTTPServer,
268- _transport=_urllib.HttpTransport_urllib,)),
269+ _transport=_urllib.HttpTransport_urllib,)),
270 ]
271 if tests.HTTPSServerFeature.available():
272 activity_scenarios.append(
273 ('urllib,https', dict(_activity_server=ActivityHTTPSServer,
274- _transport=_urllib.HttpTransport_urllib,)),)
275+ _transport=_urllib.HttpTransport_urllib,)),)
276 if features.pycurl.available():
277 activity_scenarios.append(
278 ('pycurl,http', dict(_activity_server=ActivityHTTPServer,
279- _transport=PyCurlTransport,)),)
280+ _transport=PyCurlTransport,)),)
281 if tests.HTTPSServerFeature.available():
282 from bzrlib.tests import (
283 ssl_certs,
284@@ -181,16 +138,8 @@
285
286 activity_scenarios.append(
287 ('pycurl,https', dict(_activity_server=ActivityHTTPSServer,
288- _transport=HTTPS_pycurl_transport,)),)
289-
290- tpact_scenarios = tests.multiply_scenarios(activity_scenarios,
291- protocol_scenarios)
292- tests.multiply_tests(tpact_tests, tpact_scenarios, result)
293-
294- # No parametrization for the remaining tests
295- result.addTests(remaining_tests)
296-
297- return result
298+ _transport=HTTPS_pycurl_transport,)),)
299+ return activity_scenarios
300
301
302 class FakeManager(object):
303@@ -401,6 +350,8 @@
304 class TestHttpTransportUrls(tests.TestCase):
305 """Test the http urls."""
306
307+ scenarios = vary_by_http_client_implementation()
308+
309 def test_abs_url(self):
310 """Construction of absolute http URLs"""
311 t = self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')
312@@ -413,7 +364,7 @@
313
314 def test_invalid_http_urls(self):
315 """Trap invalid construction of urls"""
316- t = self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')
317+ self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')
318 self.assertRaises(errors.InvalidURL,
319 self._transport,
320 'http://http://bazaar-vcs.org/bzr/bzr.dev/')
321@@ -475,6 +426,11 @@
322 class TestHTTPConnections(http_utils.TestCaseWithWebserver):
323 """Test the http connections."""
324
325+ scenarios = multiply_scenarios(
326+ vary_by_http_client_implementation(),
327+ vary_by_http_protocol_version(),
328+ )
329+
330 def setUp(self):
331 http_utils.TestCaseWithWebserver.setUp(self)
332 self.build_tree(['foo/', 'foo/bar'], line_endings='binary',
333@@ -525,6 +481,8 @@
334 class TestHttpTransportRegistration(tests.TestCase):
335 """Test registrations of various http implementations"""
336
337+ scenarios = vary_by_http_client_implementation()
338+
339 def test_http_registered(self):
340 t = transport.get_transport('%s://foo.com/' % self._url_protocol)
341 self.assertIsInstance(t, transport.Transport)
342@@ -533,6 +491,11 @@
343
344 class TestPost(tests.TestCase):
345
346+ scenarios = multiply_scenarios(
347+ vary_by_http_client_implementation(),
348+ vary_by_http_protocol_version(),
349+ )
350+
351 def test_post_body_is_received(self):
352 server = RecordingServer(expect_body_tail='end-of-body',
353 scheme=self._url_protocol)
354@@ -585,6 +548,11 @@
355 Daughter classes are expected to override _req_handler_class
356 """
357
358+ scenarios = multiply_scenarios(
359+ vary_by_http_client_implementation(),
360+ vary_by_http_protocol_version(),
361+ )
362+
363 # Provide a useful default
364 _req_handler_class = http_server.TestingHTTPRequestHandler
365
366@@ -841,7 +809,7 @@
367 t = self.get_readonly_transport()
368 # force transport to issue multiple requests
369 t._get_max_size = 2
370- l = list(t.readv('a', ((0, 1), (1, 1), (2, 4), (6, 4))))
371+ list(t.readv('a', ((0, 1), (1, 1), (2, 4), (6, 4))))
372 # The server should have issued 3 requests
373 self.assertEqual(3, server.GET_request_nb)
374 self.assertEqual('0123456789', t.get_bytes('a'))
375@@ -924,6 +892,8 @@
376 def get_multiple_ranges(self, file, file_size, ranges):
377 self.send_response(206)
378 self.send_header('Accept-Ranges', 'bytes')
379+ # XXX: this is strange; the 'random' name below seems undefined and
380+ # yet the tests pass -- mbp 2010-10-11 bug 658773
381 boundary = "%d" % random.randint(0,0x7FFFFFFF)
382 self.send_header("Content-Type",
383 "multipart/byteranges; boundary=%s" % boundary)
384@@ -1055,6 +1025,11 @@
385 class TestLimitedRangeRequestServer(http_utils.TestCaseWithWebserver):
386 """Tests readv requests against a server erroring out on too much ranges."""
387
388+ scenarios = multiply_scenarios(
389+ vary_by_http_client_implementation(),
390+ vary_by_http_protocol_version(),
391+ )
392+
393 # Requests with more range specifiers will error out
394 range_limit = 3
395
396@@ -1134,6 +1109,11 @@
397 to the file names).
398 """
399
400+ scenarios = multiply_scenarios(
401+ vary_by_http_client_implementation(),
402+ vary_by_http_protocol_version(),
403+ )
404+
405 # FIXME: We don't have an https server available, so we don't
406 # test https connections. --vila toolongago
407
408@@ -1236,6 +1216,11 @@
409 class TestRanges(http_utils.TestCaseWithWebserver):
410 """Test the Range header in GET methods."""
411
412+ scenarios = multiply_scenarios(
413+ vary_by_http_client_implementation(),
414+ vary_by_http_protocol_version(),
415+ )
416+
417 def setUp(self):
418 http_utils.TestCaseWithWebserver.setUp(self)
419 self.build_tree_contents([('a', '0123456789')],)
420@@ -1281,6 +1266,11 @@
421 class TestHTTPRedirections(http_utils.TestCaseWithRedirectedWebserver):
422 """Test redirection between http servers."""
423
424+ scenarios = multiply_scenarios(
425+ vary_by_http_client_implementation(),
426+ vary_by_http_protocol_version(),
427+ )
428+
429 def setUp(self):
430 super(TestHTTPRedirections, self).setUp()
431 self.build_tree_contents([('a', '0123456789'),
432@@ -1349,6 +1339,11 @@
433 -- vila 20070212
434 """
435
436+ scenarios = multiply_scenarios(
437+ vary_by_http_client_implementation(),
438+ vary_by_http_protocol_version(),
439+ )
440+
441 def setUp(self):
442 if (features.pycurl.available()
443 and self._transport == PyCurlTransport):
444@@ -1399,6 +1394,11 @@
445 class TestDoCatchRedirections(http_utils.TestCaseWithRedirectedWebserver):
446 """Test transport.do_catching_redirections."""
447
448+ scenarios = multiply_scenarios(
449+ vary_by_http_client_implementation(),
450+ vary_by_http_protocol_version(),
451+ )
452+
453 def setUp(self):
454 super(TestDoCatchRedirections, self).setUp()
455 self.build_tree_contents([('a', '0123456789'),],)
456@@ -1446,6 +1446,12 @@
457 class TestAuth(http_utils.TestCaseWithWebserver):
458 """Test authentication scheme"""
459
460+ scenarios = multiply_scenarios(
461+ vary_by_http_client_implementation(),
462+ vary_by_http_protocol_version(),
463+ vary_by_http_auth_scheme(),
464+ )
465+
466 _auth_header = 'Authorization'
467 _password_prompt_prefix = ''
468 _username_prompt_prefix = ''
469@@ -1655,6 +1661,12 @@
470 class TestProxyAuth(TestAuth):
471 """Test proxy authentication schemes."""
472
473+ scenarios = multiply_scenarios(
474+ vary_by_http_client_implementation(),
475+ vary_by_http_protocol_version(),
476+ vary_by_http_proxy_auth_scheme(),
477+ )
478+
479 _auth_header = 'Proxy-authorization'
480 _password_prompt_prefix = 'Proxy '
481 _username_prompt_prefix = 'Proxy '
482@@ -1716,6 +1728,11 @@
483
484 class SmartHTTPTunnellingTest(tests.TestCaseWithTransport):
485
486+ scenarios = multiply_scenarios(
487+ vary_by_http_client_implementation(),
488+ vary_by_http_protocol_version(),
489+ )
490+
491 def setUp(self):
492 super(SmartHTTPTunnellingTest, self).setUp()
493 # We use the VFS layer as part of HTTP tunnelling tests.
494@@ -1810,6 +1827,8 @@
495
496 class Test_redirected_to(tests.TestCase):
497
498+ scenarios = vary_by_http_client_implementation()
499+
500 def test_redirected_to_subdir(self):
501 t = self._transport('http://www.example.com/foo')
502 r = t._redirected_to('http://www.example.com/foo',
503@@ -2061,6 +2080,11 @@
504
505 class TestActivity(tests.TestCase, TestActivityMixin):
506
507+ scenarios = multiply_scenarios(
508+ vary_by_http_activity(),
509+ vary_by_http_protocol_version(),
510+ )
511+
512 def setUp(self):
513 TestActivityMixin.setUp(self)
514
515@@ -2087,6 +2111,8 @@
516 class TestAuthOnRedirected(http_utils.TestCaseWithRedirectedWebserver):
517 """Test authentication on the redirected http server."""
518
519+ scenarios = vary_by_http_protocol_version()
520+
521 _auth_header = 'Authorization'
522 _password_prompt_prefix = ''
523 _username_prompt_prefix = ''
524
525=== added file 'bzrlib/tests/test_scenarios.py'
526--- bzrlib/tests/test_scenarios.py 1970-01-01 00:00:00 +0000
527+++ bzrlib/tests/test_scenarios.py 2010-10-11 22:43:41 +0000
528@@ -0,0 +1,110 @@
529+# Copyright (C) 2005-2010 Canonical Ltd
530+#
531+# This program is free software; you can redistribute it and/or modify
532+# it under the terms of the GNU General Public License as published by
533+# the Free Software Foundation; either version 2 of the License, or
534+# (at your option) any later version.
535+#
536+# This program is distributed in the hope that it will be useful,
537+# but WITHOUT ANY WARRANTY; without even the implied warranty of
538+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
539+# GNU General Public License for more details.
540+#
541+# You should have received a copy of the GNU General Public License
542+# along with this program; if not, write to the Free Software
543+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
544+
545+
546+"""Tests for generating multiple tests for scenarios."""
547+
548+from bzrlib.tests import (
549+ TestCase,
550+ TestLoader,
551+ iter_suite_tests,
552+ multiply_tests,
553+ )
554+
555+from bzrlib.tests.scenarios import (
556+ load_tests_apply_scenarios,
557+ multiply_scenarios,
558+ multiply_tests_by_their_scenarios,
559+ )
560+
561+
562+# There aren't any actually parameterized tests here, but this exists as a
563+# demonstration; so that you can interactively observe them being multiplied;
564+# and so that we check everything hooks up properly.
565+load_tests = load_tests_apply_scenarios
566+
567+
568+def vary_by_color():
569+ """Very simple static variation example"""
570+ for color in ['red', 'green', 'blue']:
571+ yield (color, {'color': color})
572+
573+
574+def vary_named_attribute(attr_name):
575+ """More sophisticated: vary a named parameter"""
576+ yield ('a', {attr_name: 'a'})
577+ yield ('b', {attr_name: 'b'})
578+
579+
580+def get_generated_test_attributes(suite, attr_name):
581+ """Return the `attr_name` attribute from all tests in the suite"""
582+ return sorted([
583+ getattr(t, attr_name) for t in iter_suite_tests(suite)])
584+
585+
586+class TestTestScenarios(TestCase):
587+
588+ def test_multiply_tests(self):
589+ loader = TestLoader()
590+ suite = loader.suiteClass()
591+ multiply_tests(
592+ self,
593+ vary_by_color(),
594+ suite)
595+ self.assertEquals(
596+ ['blue', 'green', 'red'],
597+ get_generated_test_attributes(suite, 'color'))
598+
599+ def test_multiply_scenarios_from_generators(self):
600+ """It's safe to multiply scenarios that come from generators"""
601+ s = multiply_scenarios(
602+ vary_named_attribute('one'),
603+ vary_named_attribute('two'),
604+ )
605+ self.assertEquals(
606+ 2*2,
607+ len(s),
608+ s)
609+
610+ def test_multiply_tests_by_their_scenarios(self):
611+ loader = TestLoader()
612+ suite = loader.suiteClass()
613+ test_instance = PretendVaryingTest('test_nothing')
614+ multiply_tests_by_their_scenarios(
615+ test_instance,
616+ suite)
617+ self.assertEquals(
618+ ['a', 'a', 'b', 'b'],
619+ get_generated_test_attributes(suite, 'value'))
620+
621+ def test_multiply_tests_no_scenarios(self):
622+ """Tests with no scenarios attribute aren't multiplied"""
623+ suite = TestLoader().suiteClass()
624+ multiply_tests_by_their_scenarios(self,
625+ suite)
626+ self.assertLength(1, list(iter_suite_tests(suite)))
627+
628+
629+class PretendVaryingTest(TestCase):
630+
631+ scenarios = multiply_scenarios(
632+ vary_named_attribute('value'),
633+ vary_named_attribute('other'),
634+ )
635+
636+ def test_nothing(self):
637+ """This test exists just so it can be multiplied"""
638+ pass
639
640=== modified file 'doc/developers/testing.txt'
641--- doc/developers/testing.txt 2010-10-07 07:51:54 +0000
642+++ doc/developers/testing.txt 2010-10-11 22:43:41 +0000
643@@ -811,17 +811,11 @@
644 whether a test should be added for that particular implementation,
645 or for all implementations of the interface.
646
647-The multiplication of tests for different implementations is normally
648-accomplished by overriding the ``load_tests`` function used to load tests
649-from a module. This function typically loads all the tests, then applies
650-a TestProviderAdapter to them, which generates a longer suite containing
651-all the test variations.
652-
653 See also `Per-implementation tests`_ (above).
654
655
656-Test scenarios
657---------------
658+Test scenarios and variations
659+-----------------------------
660
661 Some utilities are provided for generating variations of tests. This can
662 be used for per-implementation tests, or other cases where the same test
663@@ -832,8 +826,24 @@
664 values to which the test should be applied. The test suite should then
665 also provide a list of scenarios in which to run the tests.
666
667-Typically ``multiply_tests_from_modules`` should be called from the test
668-module's ``load_tests`` function.
669+A single *scenario* is defined by a `(name, parameter_dict)` tuple. The
670+short string name is combined with the name of the test method to form the
671+test instance name. The parameter dict is merged into the instance's
672+attributes.
673+
674+For example::
675+
676+ load_tests = load_tests_apply_scenarios
677+
678+ class TestCheckout(TestCase):
679+
680+ variations = multiply_scenarios(
681+ VaryByRepositoryFormat(),
682+ VaryByTreeFormat(),
683+ )
684+
685+The `load_tests` declaration or definition should be near the top of the
686+file so its effect can be seen.
687
688
689 Test support